host information in domU over /proc or /sys filesystem

host information in domU over /proc or /sys filesystem

[Hannes Kuehnemund]

Hi,

I have a few questions. But lets start with the background:

Several other virtualization technologies beside Xen do provide the 
possibility to get configuration information of the VM itself and the 
host it is running on through the /proc file system. These technologies 
are for example zLinux or Linux on ppc64.

This feature is very important for quality of service aspects. How many 
physical CPU do I have? How many CPU time can I use? Other important 
information would the real memory and the hostname of the host machine. 
Basically all this information is provided by other virtualization 
technologies as read only values. Having a huge hosting solution with 
enterprise software running in virtual machines depends on these quality 
of service aspects. Having a black box makes debugging of performance 
issues impossible and thus the whole solution is not usable.

Is it possible to provide such a solution with Xen (of course only if 
enabled in dom0 due to security reasons!) over the /proc or /sysfs?

Thanks,
Hannes

-- 

* Hannes Kuehnemund
* SAP Linuxlab
* SAP AG
* Dietmar Hopp Allee 16
* 69190 Walldorf, Germany
T  +49 6227 7-40615
F  +49 6227 78-34584
mailto: hannes.kuehnemund@sap.com
http://www.sap.com

Sitz der Gesellschaft/Registered Office: Walldorf, Germany
Vorstand/SAP Executive Board: Henning Kagermann (Sprecher/CEO),
Léo Apotheker (stellvertretender Sprecher/Deputy CEO),
Werner Brandt, Claus Heinrich, Gerhard Oswald, Peter Zencke
Vorsitzender des Aufsichtsrats/Chairperson of the SAP
Supervisory Board: Hasso Plattner
Registergericht/Commercial Register Mannheim No HRB 350269

Diese E-Mail kann Betriebs- oder Geschäftsgeheimnisse oder
sonstige vertrauliche Informationen enthalten. Sollten Sie
diese E-Mail irrtümlich erhalten haben, ist Ihnen eine
Kenntnisnahme des Inhalts, eine Vervielfältigung oder
Weitergabe der E-Mail ausdrücklich untersagt. Bitte
benachrichtigen Sie uns und vernichten Sie die empfangene
E-Mail. Vielen Dank.

This e-mail may contain trade secrets or privileged,
undisclosed, or otherwise confidential information. If you have
received this e-mail in error, you are hereby notified that any
review, copying, or distribution of it is strictly prohibited.
Please inform us immediately and destroy the original
transmittal. Thank you for your cooperation.

Re: host information in domU over /proc or /sys filesystem

[Mark Williamson]

> This feature is very important for quality of service aspects. How many
> physical CPU do I have? How many CPU time can I use?

I'm not exactly sure whether a domU needs to know this?  or do you mean dom0 
specifically?

The number of physical CPUs the domain's VCPUs are being scheduled across 
might change dynamically.  You can, however, change the number of VCPUs a 
domain is allocated as an indication of how many CPUs it should try to 
optimise for.  There's also /sys/hypervisor which contains some relevant 
information.

dom0 is permitted to access more detailed information about the host machine 
through a hypercall (use "xm info" to see what's available).

> Other important > information would the real memory and the hostname of the 
host machine.

I'm not sure why a domU would need to know the hostname of the host machine?  
Also, this could change during migration, so it's not a stable value.

Cheers,
Mark

> Basically all this information is provided by other virtualization
> technologies as read only values. Having a huge hosting solution with
> enterprise software running in virtual machines depends on these quality
> of service aspects. Having a black box makes debugging of performance
> issues impossible and thus the whole solution is not usable.
>
> Is it possible to provide such a solution with Xen (of course only if
> enabled in dom0 due to security reasons!) over the /proc or /sysfs?
>
> Thanks,
> Hannes



-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: host information in domU over /proc or /sys filesystem

[Hannes Kuehnemund]

Mark Williamson wrote:
 >> This feature is very important for quality of service aspects. How
 >> many physical CPU do I have? How many CPU time can I use?
> I'm not exactly sure whether a domU needs to know this?  or do you mean dom0 
> specifically?

Actually, the domU itself shouldn't know this, but maybe an application 
running in domU. Imagine the following situation:

Three party setup:

1) Software vendor
2) Hosting provider
3) Customer (using Software from 1) and a virtual machine from 2))

In case the hosting provider changes something in the setup, the 
performance decreases and Customer complains at the software vendor the 
really bad performance of the software sold. Having no chance to get at 
least number of physical cpu's assigned to domU makes problem solving 
impossible. This theoretical setup would need the interaction of a third 
party, the Hosting provider.

If there would be a possibility for the hosting provider to provide this 
information (physical CPU assigned) to a domU, reduces the costs and 
time spend for interaction getting a third party involved in a support 
call. The information which physical CPU's are assigned to a domU does 
not affect security or encapsulation, because the domU does _not_ know 
how many physical CPU's dom0 has at all or how many of them are assigned 
to other VM's.

> The number of physical CPUs the domain's VCPUs are being scheduled across 
> might change dynamically.  You can, however, change the number of VCPUs a 
> domain is allocated as an indication of how many CPUs it should try to 
> optimise for.  

That it changes dynamically is not a problem at all. We would collect 
the data, how many physical CPU's are assigned on a 5 minute base in our 
application to keep track of it. We then can check if response times 
correlate with number of physical CPU's.

> There's also /sys/hypervisor which contains some relevant 
> information.

Yes, that for the hint. Unfortunately is data is not enough :(

> dom0 is permitted to access more detailed information about the host machine 
> through a hypercall (use "xm info" to see what's available).
>> Other important > information would the real memory and the hostname of the 
> host machine.
> I'm not sure why a domU would need to know the hostname of the host machine?  
> Also, this could change during migration, so it's not a stable value.

In case nothing like the physical CPU assignment will be implemented, 
domU must know, which CIM server to query to get this kind of 
information. I know that this value will change, and because of this we 
need the information. If the domO would stay the same all the time, I 
could set a parameter in my software to query the CIM server directly, 
which wouldn't work after a migration.

The third possibility is to setup a dedicated VM which interacts between 
all available dom0's and all available domU's. But this is the last 
possible solution I'd prefer, because a third instance to manage is a 
third source of errors.

> Cheers,
> Mark

Cheers,
   Hannes

>> Basically all this information is provided by other virtualization
>> technologies as read only values. Having a huge hosting solution with
>> enterprise software running in virtual machines depends on these quality
>> of service aspects. Having a black box makes debugging of performance
>> issues impossible and thus the whole solution is not usable.
>>
>> Is it possible to provide such a solution with Xen (of course only if
>> enabled in dom0 due to security reasons!) over the /proc or /sysfs?
>>
>> Thanks,
>> Hannes

-- 

* Hannes Kuehnemund
* SAP Linuxlab
* SAP AG
* Dietmar Hopp Allee 16
* 69190 Walldorf, Germany
T  +49 6227 7-40615
F  +49 6227 78-34584
mailto: hannes.kuehnemund@sap.com
http://www.sap.com

Sitz der Gesellschaft/Registered Office: Walldorf, Germany
Vorstand/SAP Executive Board: Henning Kagermann (Sprecher/CEO),
Léo Apotheker (stellvertretender Sprecher/Deputy CEO),
Werner Brandt, Claus Heinrich, Gerhard Oswald, Peter Zencke
Vorsitzender des Aufsichtsrats/Chairperson of the SAP
Supervisory Board: Hasso Plattner
Registergericht/Commercial Register Mannheim No HRB 350269

Diese E-Mail kann Betriebs- oder Geschäftsgeheimnisse oder
sonstige vertrauliche Informationen enthalten. Sollten Sie
diese E-Mail irrtümlich erhalten haben, ist Ihnen eine
Kenntnisnahme des Inhalts, eine Vervielfältigung oder
Weitergabe der E-Mail ausdrücklich untersagt. Bitte
benachrichtigen Sie uns und vernichten Sie die empfangene
E-Mail. Vielen Dank.

This e-mail may contain trade secrets or privileged,
undisclosed, or otherwise confidential information. If you have
received this e-mail in error, you are hereby notified that any
review, copying, or distribution of it is strictly prohibited.
Please inform us immediately and destroy the original
transmittal. Thank you for your cooperation.

Re: host information in domU over /proc or /sys filesystem

[Mark Williamson]

> > The number of physical CPUs the domain's VCPUs are being scheduled across
> > might change dynamically.  You can, however, change the number of VCPUs a
> > domain is allocated as an indication of how many CPUs it should try to
> > optimise for.
>
> That it changes dynamically is not a problem at all. We would collect
> the data, how many physical CPU's are assigned on a 5 minute base in our
> application to keep track of it. We then can check if response times
> correlate with number of physical CPU's.

Hmmmm.

Well, it's perfectly possible for the hosting provider to assign an 
approximate minimum share of the machine to the domU using the credit 
scheduler.  This is specified effectively in terms of "percentage of one 
physical CPU in a period"...

So for a domain with 90% and one vCPU it would try to schedule it 90% of the 
time on whatever PCPU is free.  For a domain with 90% and 2 vCPUs it'll try 
and make sure that the time the two vCPUs get in total is the same amount.  
Domains with > 1 vCPU can use > 100% by this metric.

Anyhow, there are a number of parameters involved here: how much time in total 
the domain should get, how many vCPUs the domain has, and how the time gets 
shared between them.

Rather than trying to sample how many pCPUs the domain gets to run on, perhaps 
it would be better to monitor the total time its vCPUs are being scheduled 
*somewhere*.  Then you can just compare whether your pCPU time allowance 
corresponds to your SLA.

This way the domain still doesn't need to know which specific CPUs it gets 
scheduled on, but you can get all the information you need?

> > dom0 is permitted to access more detailed information about the host
> > machine through a hypercall (use "xm info" to see what's available).
> >
> >> Other important > information would the real memory and the hostname of
> >> the
> >
> > host machine.
> > I'm not sure why a domU would need to know the hostname of the host
> > machine? Also, this could change during migration, so it's not a stable
> > value.
>
> In case nothing like the physical CPU assignment will be implemented,
> domU must know, which CIM server to query to get this kind of
> information. I know that this value will change, and because of this we
> need the information. If the domO would stay the same all the time, I
> could set a parameter in my software to query the CIM server directly,
> which wouldn't work after a migration.
>
> The third possibility is to setup a dedicated VM which interacts between
> all available dom0's and all available domU's. But this is the last
> possible solution I'd prefer, because a third instance to manage is a
> third source of errors.

I'm still not sure why the domU would need to know the host's hostname though?  
Or is it just as a fallback if the CPU data isn't directly available?

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: host information in domU over /proc or /sys filesystem

[Hannes Kuehnemund]

Mark Williamson wrote:
> Rather than trying to sample how many pCPUs the domain gets to run on, perhaps 
> it would be better to monitor the total time its vCPUs are being scheduled 
> *somewhere*.  Then you can just compare whether your pCPU time allowance 
> corresponds to your SLA.
> 
> This way the domain still doesn't need to know which specific CPUs it gets 
> scheduled on, but you can get all the information you need?

VMware for example handles this information in a similar way. Having the 
total consumed CPU time (cpu_time) for a VM inside the VM (which is only 
available in dom0 at the moment) one can calculate how many computing 
power is available.

Lets consider, that my VM has 2 VCPUs configured. The VM furthermore is 
configured to run on 2 physical CPUs, and this is also the SLA between 
the hosting provider and the customer. But, another VM is also running 
on these 2 physical CPU's. Having the possibility to check the cpu_time 
on a frequent base, e.g. seconds, one would be able to check, if the 
guaranteed phsical CPUs are available. To check this, simply generate a 
high load inside the VM, and if the cpu_time increases every second by 
the value of 2, both physical CPUs are available. In case the cpu_time 
only increases by the value on 1 or 1,5 it is clear, that the SLA is not 
fullfilled.

Is it possbile to provide this information to the domU (if explicitly 
enabled in dom0)?

>> The third possibility is to setup a dedicated VM which interacts between
>> all available dom0's and all available domU's. But this is the last
>> possible solution I'd prefer, because a third instance to manage is a
>> third source of errors.
> I'm still not sure why the domU would need to know the host's hostname though?  
> Or is it just as a fallback if the CPU data isn't directly available?

This is meant to be the fallback, in case it is not possible to get any 
data from the hypervisor. I'm not sure, but having HVM setups running, 
getting data might be very difficult. Therefore, having a fallback 
(which can be turned on, but will be turned off by default) can be very 
convenient.

> Cheers,
> Mark

Cheers,
   Hannes

-- 

* Hannes Kuehnemund
* SAP Linuxlab
* SAP AG
* Dietmar Hopp Allee 16
* 69190 Walldorf, Germany
T  +49 6227 7-40615
F  +49 6227 78-34584
mailto: hannes.kuehnemund@sap.com
http://www.sap.com

Sitz der Gesellschaft/Registered Office: Walldorf, Germany
Vorstand/SAP Executive Board: Henning Kagermann (Sprecher/CEO),
Léo Apotheker (stellvertretender Sprecher/Deputy CEO),
Werner Brandt, Claus Heinrich, Gerhard Oswald, Peter Zencke
Vorsitzender des Aufsichtsrats/Chairperson of the SAP
Supervisory Board: Hasso Plattner
Registergericht/Commercial Register Mannheim No HRB 350269

Diese E-Mail kann Betriebs- oder Geschäftsgeheimnisse oder
sonstige vertrauliche Informationen enthalten. Sollten Sie
diese E-Mail irrtümlich erhalten haben, ist Ihnen eine
Kenntnisnahme des Inhalts, eine Vervielfältigung oder
Weitergabe der E-Mail ausdrücklich untersagt. Bitte
benachrichtigen Sie uns und vernichten Sie die empfangene
E-Mail. Vielen Dank.

This e-mail may contain trade secrets or privileged,
undisclosed, or otherwise confidential information. If you have
received this e-mail in error, you are hereby notified that any
review, copying, or distribution of it is strictly prohibited.
Please inform us immediately and destroy the original
transmittal. Thank you for your cooperation.