Problems about CPU Affinity

Problems about CPU Affinity

[Saori Fukuta]

I have two problems about CPU Affinity.

1. On boot
    We do not have the way of setting CPU Affinity to each VCPU of guest
    domain by using a configuration file.
    So we have to execute the commands after starting the guest domain.

2. On reboot
    We lost the information of setting CPU Affinity that set while the
    guest domain is running after rebooting the guest domain.
    So we have to set it again.

To solve latter problem, this is one of the patch that I need to apply.
http://lists.xensource.com/archives/html/xen-devel/2007-03/msg01295.html

Thanks,
Saori Fukuta

RE: Problems about CPU Affinity

[Ian Pratt]

> 1. On boot
>     We do not have the way of setting CPU Affinity to each VCPU of
guest
>     domain by using a configuration file.
>     So we have to execute the commands after starting the guest
domain.

I thought you could by specifying a list of physical CPUs? Perhaps this
got broken by the xapi changes?
 
> 2. On reboot
>     We lost the information of setting CPU Affinity that set while the
>     guest domain is running after rebooting the guest domain.
>     So we have to set it again.

Affinity information should certainly be persistent.

Best,
Ian

RE: Problems about CPU Affinity

[Petersson, Mats]

 

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Ian Pratt
> Sent: 18 May 2007 10:40
> To: Saori Fukuta; xen-devel
> Subject: RE: [Xen-devel] Problems about CPU Affinity
> 
> > 1. On boot
> >     We do not have the way of setting CPU Affinity to each VCPU of
> guest
> >     domain by using a configuration file.
> >     So we have to execute the commands after starting the guest
> domain.
> 
> I thought you could by specifying a list of physical CPUs? 
> Perhaps this
> got broken by the xapi changes?

No, the problem that the original post talks about, is that you can't
set affinity of individual VCPU's to individual PCPU's, e.g:
vcpus=2
cpus=2
Works well, but there's no way to perform:
vcpu0=pcpu0
vcpu1=pcpu1
Other than running "vm vcpu-set <domain> <vcpu> <pcpu(s)>" after the
domain was started. 

--
Mats
>  
> > 2. On reboot
> >     We lost the information of setting CPU Affinity that 
> set while the
> >     guest domain is running after rebooting the guest domain.
> >     So we have to set it again.
> 
> Affinity information should certainly be persistent.
> 
> Best,
> Ian
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
> 

Re: Problems about CPU Affinity

[Atsushi SAKAI]

Hi, Ian 

Thank you for your comments.
Detailed description is written by her in following mail. 
http://lists.xensource.com/archives/html/xen-devel/2007-05/msg00416.html
She is out now. I reply it instead.

Thanks
Atsushi SAKAI

"Ian Pratt" <Ian.Pratt@cl.cam.ac.uk> wrote:

> > 1. On boot
> >     We do not have the way of setting CPU Affinity to each VCPU of
> guest
> >     domain by using a configuration file.
> >     So we have to execute the commands after starting the guest
> domain.
> 
> I thought you could by specifying a list of physical CPUs? Perhaps this
> got broken by the xapi changes?

Cannot specify pcpu-vcpu one-to-one on boot time is her problem.
I think this is not coming from xapi changes.

>  
> > 2. On reboot
> >     We lost the information of setting CPU Affinity that set while the
> >     guest domain is running after rebooting the guest domain.
> >     So we have to set it again.
> 
> Affinity information should certainly be persistent.
I hope so
> 
> Best,
> Ian
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: Problems about CPU Affinity

[Ian Pratt]

> > I thought you could by specifying a list of physical CPUs?
> > Perhaps this
> > got broken by the xapi changes?
> 
> No, the problem that the original post talks about, is that you can't
> set affinity of individual VCPU's to individual PCPU's, e.g:
> vcpus=2
> cpus=2

AFAIK, you used to be able to specify a list:
 vcpus=2
 cpus = ["2,3,4","5,6,7"]

Maybe this patch never went in. AFAIR it was from Ryan. 

Ian

Re: Problems about CPU Affinity

[Masaki Kanno]

[-- Attachment #1: Mail message body --]
[-- Type: text/plain, Size: 767 bytes --]


>I have two problems about CPU Affinity.
>
>1. On boot
>    We do not have the way of setting CPU Affinity to each VCPU of guest
>    domain by using a configuration file.
>    So we have to execute the commands after starting the guest domain.
>
>2. On reboot
>    We lost the information of setting CPU Affinity that set while the
>    guest domain is running after rebooting the guest domain.
>    So we have to set it again.
>
>To solve latter problem, this is one of the patch that I need to apply.
>http://lists.xensource.com/archives/html/xen-devel/2007-03/msg01295.html

Hi,

The patch cannot solve the problem 1, but it can solve the bug#936.
I resend it.  Could you apply it?

Signed-off-by: Masaki Kanno <kanno.masaki@jp.fujitsu.com>

Best regards,
 Kan


[-- Attachment #2: bug936.patch --]
[-- Type: application/octet-stream, Size: 3735 bytes --]

diff -r 089696e0c603 tools/python/xen/xend/XendConfig.py
--- a/tools/python/xen/xend/XendConfig.py	Thu May 17 11:42:46 2007 +0100
+++ b/tools/python/xen/xend/XendConfig.py	Fri May 18 19:25:13 2007 +0900
@@ -587,30 +587,46 @@ class XendConfig(dict):
             else:
                 cfg['cpus'] = str(cfg['cpu'])
 
-        # convert 'cpus' string to list of ints
-        # 'cpus' supports a list of ranges (0-3), seperated by
-        # commas, and negation, (^1).  
-        # Precedence is settled by  order of the string:
-        #     "0-3,^1"   -> [0,2,3]
-        #     "0-3,^1,1" -> [0,1,2,3]
-        try:
-            if 'cpus' in cfg and type(cfg['cpus']) != list:
-                cpus = []
-                for c in cfg['cpus'].split(','):
-                    if c.find('-') != -1:             
-                        (x, y) = c.split('-')
-                        for i in range(int(x), int(y)+1):
-                            cpus.append(int(i))
-                    else:
-                        # remove this element from the list 
-                        if c[0] == '^':
-                            cpus = [x for x in cpus if x != int(c[1:])]
+        # Convert 'cpus' to list of ints
+        if 'cpus' in cfg:
+            cpus = []
+            if type(cfg['cpus']) == list:
+                # If sxp_cfg was created from config.sxp,
+                # the form of 'cpus' is list of string.
+                # Convert 'cpus' to list of ints.
+                #    ['1']         -> [1]
+                #    ['0','2','3'] -> [0,2,3]
+                try:
+                    for c in cfg['cpus']:
+                        cpus.append(int(c))
+                    
+                    cfg['cpus'] = cpus
+                except ValueError, e:
+                    raise XendConfigError('cpus = %s: %s' % (cfg['cpus'], e))
+            else:
+                # Convert 'cpus' string to list of ints
+                # 'cpus' supports a list of ranges (0-3),
+                # seperated by commas, and negation, (^1).  
+                # Precedence is settled by order of the 
+                # string:
+                #    "0-3,^1"      -> [0,2,3]
+                #    "0-3,^1,1"    -> [0,1,2,3]
+                try:
+                    for c in cfg['cpus'].split(','):
+                        if c.find('-') != -1:             
+                            (x, y) = c.split('-')
+                            for i in range(int(x), int(y)+1):
+                                cpus.append(int(i))
                         else:
-                            cpus.append(int(c))
-
-                cfg['cpus'] = cpus
-        except ValueError, e:
-            raise XendConfigError('cpus = %s: %s' % (cfg['cpus'], e))
+                            # remove this element from the list 
+                            if c[0] == '^':
+                                cpus = [x for x in cpus if x != int(c[1:])]
+                            else:
+                                cpus.append(int(c))
+                    
+                    cfg['cpus'] = cpus
+                except ValueError, e:
+                    raise XendConfigError('cpus = %s: %s' % (cfg['cpus'], e))
 
         if 'security' in cfg and isinstance(cfg['security'], str):
             cfg['security'] = sxp.from_string(cfg['security'])
@@ -842,6 +858,8 @@ class XendConfig(dict):
                 if name in self and self[name] not in (None, []):
                     if typ == dict:
                         s = self[name].items()
+                    elif typ == list:
+                        s = self[name]
                     else:
                         s = str(self[name])
                     sxpr.append([name, s])

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Problems about CPU Affinity

[Masaki Kanno]

>> > I thought you could by specifying a list of physical CPUs?
>> > Perhaps this
>> > got broken by the xapi changes?
>> 
>> No, the problem that the original post talks about, is that you can't
>> set affinity of individual VCPU's to individual PCPU's, e.g:
>> vcpus=2
>> cpus=2
>
>AFAIK, you used to be able to specify a list:
> vcpus=2
> cpus = ["2,3,4","5,6,7"]

Hi,

I tried it. It does not work.

# xm create cpus=["2,3,4","5,6,7"] vcpus=2 ./PVdomain.1
Using config file "././PVdomain.1".
Error: cpus = [2,3,4,5,6,7]: invalid literal for int(): [2
# xm create cpus="["2,3,4","5,6,7"]" vcpus=2 ./PVdomain.1
Using config file "././PVdomain.1".
Error: cpus = [2,3,4,5,6,7]: invalid literal for int(): [2

Best regards,
 Kan

RE: Problems about CPU Affinity

[Masaki Kanno]

>
>>> > I thought you could by specifying a list of physical CPUs?
>>> > Perhaps this
>>> > got broken by the xapi changes?
>>> 
>>> No, the problem that the original post talks about, is that you can't
>>> set affinity of individual VCPU's to individual PCPU's, e.g:
>>> vcpus=2
>>> cpus=2
>>
>>AFAIK, you used to be able to specify a list:
>> vcpus=2
>> cpus = ["2,3,4","5,6,7"]
>
>Hi,
>
>I tried it. It does not work.
>
># xm create cpus=["2,3,4","5,6,7"] vcpus=2 ./PVdomain.1
>Using config file "././PVdomain.1".
>Error: cpus = [2,3,4,5,6,7]: invalid literal for int(): [2
># xm create cpus="["2,3,4","5,6,7"]" vcpus=2 ./PVdomain.1
>Using config file "././PVdomain.1".
>Error: cpus = [2,3,4,5,6,7]: invalid literal for int(): [2
>

Hi Ian,

Sorry, it was my mistake.  I thought that the cpus parameter 
is able to specify a list currently. 

Could you tell me why the patch was not applied?

Best regards,
 Kan

Re: Problems about CPU Affinity

[Ryan Harper]

* Ian Pratt <Ian.Pratt@cl.cam.ac.uk> [2007-05-18 05:36]:
> > > I thought you could by specifying a list of physical CPUs?
> > > Perhaps this
> > > got broken by the xapi changes?
> > 
> > No, the problem that the original post talks about, is that you can't
> > set affinity of individual VCPU's to individual PCPU's, e.g:
> > vcpus=2
> > cpus=2
> 
> AFAIK, you used to be able to specify a list:
>  vcpus=2
>  cpus = ["2,3,4","5,6,7"]
> 
> Maybe this patch never went in. AFAIR it was from Ryan. 

I don't think we ever took the multi cpumask per vcpu patch.  At the
time, only the part where we apply the same cpumask defined from cpus=""
value to each vcpu when in allowing the credit scheduler to balance within
the affinity mask.

Couple of threads of discussion here:

http://lists.xensource.com/archives/html/xen-devel/2006-08/msg00965.html

follow up threads:

http://lists.xensource.com/archives/html/xen-devel/2006-08/msg00783.html
http://lists.xensource.com/archives/html/xen-devel/2006-08/msg00785.html

The only changes that is see committed were:

changeset:   11111:438ed1c4b391
user:        kfraser@localhost.localdomain
date:        Tue Aug 15 10:56:59 2006 +0100
summary:     [XEND] Apply the domain cpumask fully to every vcpu in the domain.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: Problems about CPU Affinity

[Saori Fukuta]

Hi,

Thank you for your comments !

On Fri, 18 May 2007 10:07:46 -0500 Ryan Harper wrote:
> * Ian Pratt <Ian.Pratt@cl.cam.ac.uk> [2007-05-18 05:36]:
> > > > I thought you could by specifying a list of physical CPUs?
> > > > Perhaps this
> > > > got broken by the xapi changes?
> > > 
> > > No, the problem that the original post talks about, is that you can't
> > > set affinity of individual VCPU's to individual PCPU's, e.g:
> > > vcpus=2
> > > cpus=2
> > 
> > AFAIK, you used to be able to specify a list:
> >  vcpus=2
> >  cpus = ["2,3,4","5,6,7"]

Yes, I was talking about the things described above.

> > Maybe this patch never went in. AFAIR it was from Ryan. 
>
> I don't think we ever took the multi cpumask per vcpu patch.  At the
> time, only the part where we apply the same cpumask defined from cpus=""
> value to each vcpu when in allowing the credit scheduler to balance within
> the affinity mask.
> 
> Couple of threads of discussion here:
> 
> http://lists.xensource.com/archives/html/xen-devel/2006-08/msg00965.html
> 
> The only changes that is see committed were:
> 
> changeset:   11111:438ed1c4b391
> user:        kfraser@localhost.localdomain
> date:        Tue Aug 15 10:56:59 2006 +0100
> summary:     [XEND] Apply the domain cpumask fully to every vcpu in the domain.

Therefore, I can not specify such a list now.

If everything is OK, could you apply these patches ?

Thanks,
Saori Fukuta