domains being migrated state message improvements

domains being migrated state message improvements

[Stefan Parvu]

Hi,

Can anyone comment and consider improving the way Xen kernel reports
domains being migrated ? In short Im putting some load on 10-14 guests
and I want to monitor things from dom0 via xentop. Im seeing strange
things reported by xentop like '------'

See entire thread here:
http://lists.xen.org/archives/html/xen-users/2014-05/msg00007.html


Would be useful to have a proper message reported by xl list and xentop
when domains are being migrated. Currently xentop and xl list reporting
things like:

xentop:

>       NAME  STATE   CPU(sec) CPU(%)    
>      c5932 ------       1066   53.1    
>      c5964 -----r       3439   50.7    
>      c6464 ------       1067   50.3    
>    deb7464 ------        630   52.1    
>   Domain-0 -----r      82853  111.5    
>      lobby ------       1380   52.2    
>      r5732 ------      12075   50.7    
>      r5764 -----r      24563   50.6    
>      s10u8 -----r       2647   47.6    
>     sdrcom -----r       1015   52.7    
>     sdrorg --b---        154    0.0    
>    u100432 ------        961   50.9    
>    u100464 -----r        978   52.2    
>    u120464 ------        969   52.7    
>     win764 -----r       1328   72.7  


xl list:

Name                   ID   Mem VCPUs   State   Time(s)
Domain-0               0  6500     2     r-----   87994.3
c5932                  1  1024     1     r-----    8998.7
c5964                  2  1024     1     ------    6786.1
c6464                  3  1024     1     r-----    1896.2
deb7464                4  1024     1     ------     792.3
lobby                  5   756     1     ------    2403.7
r5732                  6  1019     1     r-----   27343.8
r5764                  7  1019     1     ------   27140.4
s10u8                  8  1019     1     r-----    3583.6
sdrcom                 9   768     1     r-----    1852.4
sdrorg                 10   768     1     -b----     168.6     
u100432                11  1024     1     ------   15657.9     
u100464                12  1024     1     r-----    1805.2     
u120464                13  1024     1     r-----    1812.1     
win764                 15  1023     2     ------    3131.6  


Please consider enhancing the output and have proper logging of each domain for
different conditions. Output like '------' is confusing and does not describe
properly the things. Worse 3rd tools or consumers of xentop might break 
when receiving such output.  

Thanks,

-- 
Stefan Parvu <sparvu@systemdatarecorder.org>

Re: domains being migrated state message improvements

[Ian Campbell]

On Fri, 2014-05-02 at 16:16 +0300, Stefan Parvu wrote:
> Please consider enhancing the output and have proper logging of each domain for
> different conditions. Output like '------' is confusing and does not describe
> properly the things. Worse 3rd tools or consumers of xentop might break 
> when receiving such output.  

Patches will be gratefully received.

Ian.

Re: domains being migrated state message improvements

[Stefan Parvu]

On Fri, 2014-05-02 at 14:23 +0100, Ian Campbell wrote:
> On Fri, 2014-05-02 at 16:16 +0300, Stefan Parvu wrote:
> > Please consider enhancing the output and have proper logging of each domain for
> > different conditions. Output like '------' is confusing and does not describe
> > properly the things. Worse 3rd tools or consumers of xentop might break 
> > when receiving such output.  
> 
> Patches will be gratefully received.

Im not 100% fluent in C but I will try as my time permits me. Do you
guys use any bugzilla or any other type of defect mgmt to report such
issues, that they will stay logged ?


-- 
Stefan Parvu <sparvu@systemdatarecorder.org>

Re: domains being migrated state message improvements

[Ian Campbell]

On Fri, 2014-05-02 at 16:31 +0300, Stefan Parvu wrote:
> On Fri, 2014-05-02 at 14:23 +0100, Ian Campbell wrote:
> > On Fri, 2014-05-02 at 16:16 +0300, Stefan Parvu wrote:
> > > Please consider enhancing the output and have proper logging of each domain for
> > > different conditions. Output like '------' is confusing and does not describe
> > > properly the things. Worse 3rd tools or consumers of xentop might break 
> > > when receiving such output.  
> > 
> > Patches will be gratefully received.
> 
> Im not 100% fluent in C but I will try as my time permits me. Do you
> guys use any bugzilla or any other type of defect mgmt to report such
> issues, that they will stay logged ?

http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen

Ian.

Re: domains being migrated state message improvements

[Stefan Parvu]

> 
> http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen

Thank you. I opened Bug id: 1888 for future references.


-- 
Stefan Parvu <sparvu@systemdatarecorder.org>

Re: domains being migrated state message improvements

[Andrew Cooper]

On 02/05/14 14:16, Stefan Parvu wrote:
> Hi,
>
> Can anyone comment and consider improving the way Xen kernel reports
> domains being migrated ? In short Im putting some load on 10-14 guests
> and I want to monitor things from dom0 via xentop. Im seeing strange
> things reported by xentop like '------'
>
> See entire thread here:
> http://lists.xen.org/archives/html/xen-users/2014-05/msg00007.html
>
>
> Would be useful to have a proper message reported by xl list and xentop
> when domains are being migrated. Currently xentop and xl list reporting
> things like:
>
> xentop:
>
>>       NAME  STATE   CPU(sec) CPU(%)    
>>      c5932 ------       1066   53.1    
>>      c5964 -----r       3439   50.7    
>>      c6464 ------       1067   50.3    
>>    deb7464 ------        630   52.1    
>>   Domain-0 -----r      82853  111.5    
>>      lobby ------       1380   52.2    
>>      r5732 ------      12075   50.7    
>>      r5764 -----r      24563   50.6    
>>      s10u8 -----r       2647   47.6    
>>     sdrcom -----r       1015   52.7    
>>     sdrorg --b---        154    0.0    
>>    u100432 ------        961   50.9    
>>    u100464 -----r        978   52.2    
>>    u120464 ------        969   52.7    
>>     win764 -----r       1328   72.7  
>
> xl list:
>
> Name                   ID   Mem VCPUs   State   Time(s)
> Domain-0               0  6500     2     r-----   87994.3
> c5932                  1  1024     1     r-----    8998.7
> c5964                  2  1024     1     ------    6786.1
> c6464                  3  1024     1     r-----    1896.2
> deb7464                4  1024     1     ------     792.3
> lobby                  5   756     1     ------    2403.7
> r5732                  6  1019     1     r-----   27343.8
> r5764                  7  1019     1     ------   27140.4
> s10u8                  8  1019     1     r-----    3583.6
> sdrcom                 9   768     1     r-----    1852.4
> sdrorg                 10   768     1     -b----     168.6     
> u100432                11  1024     1     ------   15657.9     
> u100464                12  1024     1     r-----    1805.2     
> u120464                13  1024     1     r-----    1812.1     
> win764                 15  1023     2     ------    3131.6  
>
>
> Please consider enhancing the output and have proper logging of each domain for
> different conditions. Output like '------' is confusing and does not describe
> properly the things. Worse 3rd tools or consumers of xentop might break 
> when receiving such output.  
>
> Thanks,
>

I would agree that it is a little confusing.  '------' means runnable
but not running, in this case due to vcpu over subscription.

However, the information is stale by the time it is displayed.  It is
not really appropriate to be consumed by anything other than an eyeball.

~Andrew

Re: domains being migrated state message improvements

[Ian Campbell]

On Fri, 2014-05-02 at 14:24 +0100, Andrew Cooper wrote:
> On 02/05/14 14:16, Stefan Parvu wrote:
> > Hi,
> >
> > Can anyone comment and consider improving the way Xen kernel reports
> > domains being migrated ? In short Im putting some load on 10-14 guests
> > and I want to monitor things from dom0 via xentop. Im seeing strange
> > things reported by xentop like '------'
> >
> > See entire thread here:
> > http://lists.xen.org/archives/html/xen-users/2014-05/msg00007.html
> >
> >
> > Would be useful to have a proper message reported by xl list and xentop
> > when domains are being migrated. Currently xentop and xl list reporting
> > things like:
> >
> > xentop:
> >
> >>       NAME  STATE   CPU(sec) CPU(%)    
> >>      c5932 ------       1066   53.1    
> >>      c5964 -----r       3439   50.7    
> >>      c6464 ------       1067   50.3    
> >>    deb7464 ------        630   52.1    
> >>   Domain-0 -----r      82853  111.5    
> >>      lobby ------       1380   52.2    
> >>      r5732 ------      12075   50.7    
> >>      r5764 -----r      24563   50.6    
> >>      s10u8 -----r       2647   47.6    
> >>     sdrcom -----r       1015   52.7    
> >>     sdrorg --b---        154    0.0    
> >>    u100432 ------        961   50.9    
> >>    u100464 -----r        978   52.2    
> >>    u120464 ------        969   52.7    
> >>     win764 -----r       1328   72.7  
> >
> > xl list:
> >
> > Name                   ID   Mem VCPUs   State   Time(s)
> > Domain-0               0  6500     2     r-----   87994.3
> > c5932                  1  1024     1     r-----    8998.7
> > c5964                  2  1024     1     ------    6786.1
> > c6464                  3  1024     1     r-----    1896.2
> > deb7464                4  1024     1     ------     792.3
> > lobby                  5   756     1     ------    2403.7
> > r5732                  6  1019     1     r-----   27343.8
> > r5764                  7  1019     1     ------   27140.4
> > s10u8                  8  1019     1     r-----    3583.6
> > sdrcom                 9   768     1     r-----    1852.4
> > sdrorg                 10   768     1     -b----     168.6     
> > u100432                11  1024     1     ------   15657.9     
> > u100464                12  1024     1     r-----    1805.2     
> > u120464                13  1024     1     r-----    1812.1     
> > win764                 15  1023     2     ------    3131.6  
> >
> >
> > Please consider enhancing the output and have proper logging of each domain for
> > different conditions. Output like '------' is confusing and does not describe
> > properly the things. Worse 3rd tools or consumers of xentop might break 
> > when receiving such output.  
> >
> > Thanks,
> >
> 
> I would agree that it is a little confusing.  '------' means runnable
> but not running, in this case due to vcpu over subscription.

Is that not "blocked"?

BTW, the "migration" referred to here is VCPU migration to another PCPU
rather than domain migration. (It came up because that's another one of
the VCPU pause flags alongside blocked et al)

> However, the information is stale by the time it is displayed.  It is
> not really appropriate to be consumed by anything other than an eyeball.

Agreed. And even if someone disagrees I think it is up to those tools to
accept whatever the output is, even if that includes "nonsensical"
results.

Ian.

Re: domains being migrated state message improvements

[Andrew Cooper]

On 02/05/14 14:29, Ian Campbell wrote:
>
>> I would agree that it is a little confusing.  '------' means runnable
>> but not running, in this case due to vcpu over subscription.
> Is that not "blocked"?

Blocked is any vcpu/domain pause count.  For healthy domains, this is
usually a yielded timeslice, which is why idle domains are almost always
'-b----'.

>
> BTW, the "migration" referred to here is VCPU migration to another PCPU
> rather than domain migration. (It came up because that's another one of
> the VCPU pause flags alongside blocked et al)

I do not think vcpu motion around the system is actually relevant to the
problem described.

~Andrew

>
>> However, the information is stale by the time it is displayed.  It is
>> not really appropriate to be consumed by anything other than an eyeball.
> Agreed. And even if someone disagrees I think it is up to those tools to
> accept whatever the output is, even if that includes "nonsensical"
> results.
>
> Ian.
>

Re: domains being migrated state message improvements

[Ian Campbell]

On Fri, 2014-05-02 at 14:40 +0100, Andrew Cooper wrote:
> On 02/05/14 14:29, Ian Campbell wrote:
> >
> >> I would agree that it is a little confusing.  '------' means runnable
> >> but not running, in this case due to vcpu over subscription.
> > Is that not "blocked"?
> 
> Blocked is any vcpu/domain pause count.  For healthy domains, this is
> usually a yielded timeslice, which is why idle domains are almost always
> '-b----'.
> 
> >
> > BTW, the "migration" referred to here is VCPU migration to another PCPU
> > rather than domain migration. (It came up because that's another one of
> > the VCPU pause flags alongside blocked et al)
> 
> I do not think vcpu motion around the system is actually relevant to the
> problem described.

It was due to this code in getdomaininfo:
    /* 
     * - domain is marked as blocked only if all its vcpus are blocked
     * - domain is marked as running if any of its vcpus is running
     */
    for_each_vcpu ( d, v )
    {
        vcpu_runstate_get(v, &runstate);
        cpu_time += runstate.time[RUNSTATE_running];
        info->max_vcpu_id = v->vcpu_id;
        if ( !test_bit(_VPF_down, &v->pause_flags) )
        {
            if ( !(v->pause_flags & VPF_blocked) )
                flags &= ~XEN_DOMINF_blocked;
            if ( v->is_running )
                flags |= XEN_DOMINF_running;
            info->nr_online_vcpus++;
        }
    }

If a domain was not running or blocked then it could end up with neither
set, I thought this might be when some other VPF was applied, like
VPF_migrating (or VPF_mem_paging etc).

Or are you saying that those always come along with a VPF_blocked?

Ian.

Re: domains being migrated state message improvements

[Stefan Parvu]

> > However, the information is stale by the time it is displayed.  It is
> > not really appropriate to be consumed by anything other than an eyeball.
> 
> Agreed. And even if someone disagrees I think it is up to those tools to
> accept whatever the output is, even if that includes "nonsensical"
> results.

So why do we have xentop then ? Or other tools ? To consume your
internal data, and more or less to ensure it is legit.

Remember, that we might have cases where people *are* consuming xentop
results. Or other tool(s), based on results coming from Xen kernel. 

So if I want to store 1 year statistics of all guests, as time series, I
expect to see more or less reliable data. Right ? Analyzing data will
turn in a mess.

-- 
Stefan Parvu <sparvu@systemdatarecorder.org>

Re: domains being migrated state message improvements

[Andrew Cooper]

On 02/05/14 16:47, Stefan Parvu wrote:
>>> However, the information is stale by the time it is displayed.  It is
>>> not really appropriate to be consumed by anything other than an eyeball.
>> Agreed. And even if someone disagrees I think it is up to those tools to
>> accept whatever the output is, even if that includes "nonsensical"
>> results.
> So why do we have xentop then ?

So a human can take a look at a system and see if anything looks
obviously wrong.

>  Or other tools ? To consume your
> internal data, and more or less to ensure it is legit.
>
> Remember, that we might have cases where people *are* consuming xentop
> results.

How? it uses ncurses to be interactive on the terminal.  Collecting this
output with automated tools is impractical.

>  Or other tool(s), based on results coming from Xen kernel. 

This is a bigger issue.  The values are exposed in libxc and somewhat
more statically to the shell via xl

>
> So if I want to store 1 year statistics of all guests, as time series, I
> expect to see more or less reliable data. Right ? Analyzing data will
> turn in a mess.
>

What are you looking to collect?  It surely isn't direct scheduling data.

~Andrew

Re: domains being migrated state message improvements

[Stefan Parvu]

> So a human can take a look at a system and see if anything looks
> obviously wrong.

and in our case, it is indeed wrong, or ... confusing - Man page says
what we should expect from the state field. But instead we see ------
How one should interpret that ? Even interactively :) vanished, not
available, what really happened there ?! 

First of all this should be documented and explained somewhere.


> How? it uses ncurses to be interactive on the terminal.  Collecting this
> output with automated tools is impractical.

you put xentop on batch mode to record data for long periods of time. 
store it and analyze it. 

> What are you looking to collect?  It surely isn't direct scheduling data.
> 

I want to see from dom0 more or less what xentop sees for each guest.
See all metrics documented here:
http://www.systemdatarecorder.org/recording/xenrec.html

I want to consume it via Perl, if available, that would be sweet. I did
not know how to do that and use xentop as a temporary solution. In the
long run I would like to write my won recorder to do that.

Thanks a lot,

-- 
Stefan Parvu <sparvu@systemdatarecorder.org>

Re: domains being migrated state message improvements

[Andrew Cooper]

On 02/05/14 17:46, Stefan Parvu wrote:
>> So a human can take a look at a system and see if anything looks
>> obviously wrong.
> and in our case, it is indeed wrong, or ... confusing - Man page says
> what we should expect from the state field.

Which manpage?

>  But instead we see ------
> How one should interpret that ? Even interactively :) vanished, not
> available, what really happened there ?! 
>
> First of all this should be documented and explained somewhere.

It is.  That would be "not running and not blocked" which is a valid
state for a VM to be in.

The comment in Xen still appears accurate:

    /*
     * - domain is marked as blocked only if all its vcpus are blocked
     * - domain is marked as running if any of its vcpus is running
     */

Blocked gets set whenever a vcpu uses a SCHEDOP_block or SCHEDOP_poll
hypercall.  The former for blocking until any event arrives, the latter
when waiting for a specific event.  SCHEDOP_block is used by guests when
idle.

With vcpu oversubscription, it is easy to have none of the vcpus running
at the moments when their state are sampled.  It is also easy for a
completely idle guest to have all vcpus blocked.

As a result, all 4 possible states of blocked/not blocked/running/not
running are valid.

Not blocked and not running therefore means "domain is not idle, but not
scheduled" which completely matches your description of oversubscribed
host with busy VMs.


>
>
>> How? it uses ncurses to be interactive on the terminal.  Collecting this
>> output with automated tools is impractical.
> you put xentop on batch mode to record data for long periods of time. 
> store it and analyze it. 

Wow - you learn something every day.  I was completely unaware of xentop
batch mode.

~Andrew

Re: domains being migrated state message improvements

[Stefan Parvu]

> Which manpage?

xentop man page:

AUTHORS
       Written by Judy Fischbach, David Hendricks, and Josh Triplett

REPORTING BUGS
       Report bugs to <xen-devel@lists.xen.org>.

COPYRIGHT
       Copyright © 2005  International Business Machines  Corp
       This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.


But indeed checking again I see the man page is not even listing what
metrics xentop is reporting. Uauu ... Im mainly here talking about
xentop consumer and its documentation.


> It is.  That would be "not running and not blocked" which is a valid
> state for a VM to be in.
> 
> The comment in Xen still appears accurate:
> 
>     /*
>      * - domain is marked as blocked only if all its vcpus are blocked
>      * - domain is marked as running if any of its vcpus is running
>      */
> 
> Blocked gets set whenever a vcpu uses a SCHEDOP_block or SCHEDOP_poll
> hypercall.  The former for blocking until any event arrives, the latter
> when waiting for a specific event.  SCHEDOP_block is used by guests when
> idle.
> 
> With vcpu oversubscription, it is easy to have none of the vcpus running
> at the moments when their state are sampled.  It is also easy for a
> completely idle guest to have all vcpus blocked.
> 
> As a result, all 4 possible states of blocked/not blocked/running/not
> running are valid.
> 
> Not blocked and not running therefore means "domain is not idle, but not
> scheduled" which completely matches your description of oversubscribed
> host with busy VMs.
> 

And how '------' would translate to :) ? 

I meant we need to have a bit better documentation towards people which
are planning to use data from Xen for performance analysis, capacity
planning etc ... Digging into Xen source code would be overkill in some
cases.

Thanks for pointers and explanations. 



-- 
Stefan Parvu <sparvu@systemdatarecorder.org>




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: domains being migrated state message improvements

[Ian Campbell]

On Fri, 2014-05-02 at 20:59 +0300, Stefan Parvu wrote:
> > As a result, all 4 possible states of blocked/not blocked/running/not
> > running are valid.
> > 
> > Not blocked and not running therefore means "domain is not idle, but not
> > scheduled" which completely matches your description of oversubscribed
> > host with busy VMs.
> > 
> 
> And how '------' would translate to :) ? 

"Not blocked and not running", which is what Andrew has just described.

Ian.