All of lore.kernel.org
 help / color / mirror / Atom feed
From: Avi Kivity <avi@redhat.com>
To: Yanfei Zhang <zhangyanfei@cn.fujitsu.com>
Cc: dzickus@redhat.com, luto@mit.edu,
	masanori.yoshida.tv@hitachi.com, kvm@vger.kernel.org,
	Joerg Roedel <joerg.roedel@amd.com>,
	mtosatti@redhat.com, kexec@lists.infradead.org,
	linux-kernel@vger.kernel.org, paul.gortmaker@windriver.com,
	ludwig.nussel@suse.de, ebiederm@xmission.com,
	Greg KH <gregkh@linuxfoundation.org>
Subject: Re: [PATCH v2 0/5] Export offsets of VMCS fields as note information for kdump
Date: Thu, 14 Jun 2012 16:15:43 +0300	[thread overview]
Message-ID: <4FD9E3FF.4050906@redhat.com> (raw)
In-Reply-To: <4FD58399.4050700@cn.fujitsu.com>

On 06/11/2012 08:35 AM, Yanfei Zhang wrote:
> Hello Avi,


Sorry about the delay...

> 
> 于 2012年05月29日 15:06, Yanfei Zhang 写道:
>> 于 2012年05月28日 21:28, Avi Kivity 写道:
>>> On 05/28/2012 08:25 AM, Yanfei Zhang wrote:
>>>>
>>>> Dou you have any comments about this patch set?
>>>
>>> I still have a hard time understanding why it is needed.  If the host
>>> crashes, there is no reason to look at guest state; the host should
>>> survive no matter what the guest does.
>>>
>>>
>> 
>> OK. Let me summarize it.
>> 
>> 1. Why is this patch needed? (Our requirement)
>>    
>> We once came to a buggy situation: a host scheduler bug caused guest machine's
>> vcpu stopped for a long time and then led to heartbeat stop (host is still running).
>>    
>> we want to have an efficient way to make the bug analysis when we come to the similar
>> situation where guest machine doesn't work well due to something of host machine's, 
>> 
>> Because we should debug both host machine's and guest machine's sides to look for
>> the reasons, so we want to get both host machine's crash dump and guest machine's
>> crash dump at the same time when the buggy situation remains.

I would argue that there are two separate bugs here: (1) a host bug
which caused the scheduling delay (2) putting a heartbeat service on a
virtualized guests with no real time guarantees.

But I understand your situation.

>> 
>> 2. What will we do?
>>    
>> If this bug was found on customer's environment, we have two ways to avoid
>> affecting other guest machines running on the same host. First, we could do bug
>> analysis on another environment to reproduce the buggy situation; Second, we
>> could migrate other guest machines to other hosts. 

You could also use tracing (there's the latency tracer and the scheduler
tracepoints) to debug this on a live system.

>> 
>> After the buggy situation is reproduced, we panic the host *manually*.
>> Then we could use userland tools to get guest machine's crash dump from host machine's
>> with the feature provided by this patch set. Finally we could analyse them separately
>> to find which side causes the problem.
>> 
> 
> Could you please tell me your attitude towards this patch? 

I still dislike it conceptually.  But let me do a technical review of
the latest version.

> And here is a new case from the LinuxCon Japan:
> 
> Developers from Hitach are now developing a new livedump mechanism for the
> same reason as ours. They have come to the situation *many times* that guest
> machines crashed due to host's failures, in particular, under development.

This has happened to me as well, possible even more times :).  I don't
use crash dumps for debugging but different people may use different
techniques.

> So they develop this mechanism to get crash dump while retaining the buggy
> situation between host and guest machine. The difference between theirs and
> ours is whether or not to use the feature on _customer's running machine_.


-- 
error compiling committee.c: too many arguments to function



_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

WARNING: multiple messages have this Message-ID (diff)
From: Avi Kivity <avi-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
To: Yanfei Zhang <zhangyanfei-BthXqXjhjHXQFUHtdCDX3A@public.gmane.org>
Cc: dzickus-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
	luto-3s7WtUTddSA@public.gmane.org,
	masanori.yoshida.tv-FCd8Q96Dh0JBDgjK7y7TUQ@public.gmane.org,
	kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	Joerg Roedel <joerg.roedel-5C7GfCeVMHo@public.gmane.org>,
	mtosatti-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
	kexec-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	paul.gortmaker-CWA4WttNNZF54TAoqtyWWQ@public.gmane.org,
	ludwig.nussel-l3A5Bk7waGM@public.gmane.org,
	ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org,
	Greg KH
	<gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r@public.gmane.org>
Subject: Re: [PATCH v2 0/5] Export offsets of VMCS fields as note information for kdump
Date: Thu, 14 Jun 2012 16:15:43 +0300	[thread overview]
Message-ID: <4FD9E3FF.4050906@redhat.com> (raw)
In-Reply-To: <4FD58399.4050700-BthXqXjhjHXQFUHtdCDX3A@public.gmane.org>

On 06/11/2012 08:35 AM, Yanfei Zhang wrote:
> Hello Avi,


Sorry about the delay...

> 
> 于 2012年05月29日 15:06, Yanfei Zhang 写道:
>> 于 2012年05月28日 21:28, Avi Kivity 写道:
>>> On 05/28/2012 08:25 AM, Yanfei Zhang wrote:
>>>>
>>>> Dou you have any comments about this patch set?
>>>
>>> I still have a hard time understanding why it is needed.  If the host
>>> crashes, there is no reason to look at guest state; the host should
>>> survive no matter what the guest does.
>>>
>>>
>> 
>> OK. Let me summarize it.
>> 
>> 1. Why is this patch needed? (Our requirement)
>>    
>> We once came to a buggy situation: a host scheduler bug caused guest machine's
>> vcpu stopped for a long time and then led to heartbeat stop (host is still running).
>>    
>> we want to have an efficient way to make the bug analysis when we come to the similar
>> situation where guest machine doesn't work well due to something of host machine's, 
>> 
>> Because we should debug both host machine's and guest machine's sides to look for
>> the reasons, so we want to get both host machine's crash dump and guest machine's
>> crash dump at the same time when the buggy situation remains.

I would argue that there are two separate bugs here: (1) a host bug
which caused the scheduling delay (2) putting a heartbeat service on a
virtualized guests with no real time guarantees.

But I understand your situation.

>> 
>> 2. What will we do?
>>    
>> If this bug was found on customer's environment, we have two ways to avoid
>> affecting other guest machines running on the same host. First, we could do bug
>> analysis on another environment to reproduce the buggy situation; Second, we
>> could migrate other guest machines to other hosts. 

You could also use tracing (there's the latency tracer and the scheduler
tracepoints) to debug this on a live system.

>> 
>> After the buggy situation is reproduced, we panic the host *manually*.
>> Then we could use userland tools to get guest machine's crash dump from host machine's
>> with the feature provided by this patch set. Finally we could analyse them separately
>> to find which side causes the problem.
>> 
> 
> Could you please tell me your attitude towards this patch? 

I still dislike it conceptually.  But let me do a technical review of
the latest version.

> And here is a new case from the LinuxCon Japan:
> 
> Developers from Hitach are now developing a new livedump mechanism for the
> same reason as ours. They have come to the situation *many times* that guest
> machines crashed due to host's failures, in particular, under development.

This has happened to me as well, possible even more times :).  I don't
use crash dumps for debugging but different people may use different
techniques.

> So they develop this mechanism to get crash dump while retaining the buggy
> situation between host and guest machine. The difference between theirs and
> ours is whether or not to use the feature on _customer's running machine_.


-- 
error compiling committee.c: too many arguments to function



_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

WARNING: multiple messages have this Message-ID (diff)
From: Avi Kivity <avi@redhat.com>
To: Yanfei Zhang <zhangyanfei@cn.fujitsu.com>
Cc: mtosatti@redhat.com, ebiederm@xmission.com, luto@mit.edu,
	Joerg Roedel <joerg.roedel@amd.com>,
	dzickus@redhat.com, paul.gortmaker@windriver.com,
	ludwig.nussel@suse.de, linux-kernel@vger.kernel.org,
	kvm@vger.kernel.org, kexec@lists.infradead.org,
	Greg KH <gregkh@linuxfoundation.org>,
	masanori.yoshida.tv@hitachi.com
Subject: Re: [PATCH v2 0/5] Export offsets of VMCS fields as note information for kdump
Date: Thu, 14 Jun 2012 16:15:43 +0300	[thread overview]
Message-ID: <4FD9E3FF.4050906@redhat.com> (raw)
In-Reply-To: <4FD58399.4050700@cn.fujitsu.com>

On 06/11/2012 08:35 AM, Yanfei Zhang wrote:
> Hello Avi,


Sorry about the delay...

> 
> 于 2012年05月29日 15:06, Yanfei Zhang 写道:
>> 于 2012年05月28日 21:28, Avi Kivity 写道:
>>> On 05/28/2012 08:25 AM, Yanfei Zhang wrote:
>>>>
>>>> Dou you have any comments about this patch set?
>>>
>>> I still have a hard time understanding why it is needed.  If the host
>>> crashes, there is no reason to look at guest state; the host should
>>> survive no matter what the guest does.
>>>
>>>
>> 
>> OK. Let me summarize it.
>> 
>> 1. Why is this patch needed? (Our requirement)
>>    
>> We once came to a buggy situation: a host scheduler bug caused guest machine's
>> vcpu stopped for a long time and then led to heartbeat stop (host is still running).
>>    
>> we want to have an efficient way to make the bug analysis when we come to the similar
>> situation where guest machine doesn't work well due to something of host machine's, 
>> 
>> Because we should debug both host machine's and guest machine's sides to look for
>> the reasons, so we want to get both host machine's crash dump and guest machine's
>> crash dump at the same time when the buggy situation remains.

I would argue that there are two separate bugs here: (1) a host bug
which caused the scheduling delay (2) putting a heartbeat service on a
virtualized guests with no real time guarantees.

But I understand your situation.

>> 
>> 2. What will we do?
>>    
>> If this bug was found on customer's environment, we have two ways to avoid
>> affecting other guest machines running on the same host. First, we could do bug
>> analysis on another environment to reproduce the buggy situation; Second, we
>> could migrate other guest machines to other hosts. 

You could also use tracing (there's the latency tracer and the scheduler
tracepoints) to debug this on a live system.

>> 
>> After the buggy situation is reproduced, we panic the host *manually*.
>> Then we could use userland tools to get guest machine's crash dump from host machine's
>> with the feature provided by this patch set. Finally we could analyse them separately
>> to find which side causes the problem.
>> 
> 
> Could you please tell me your attitude towards this patch? 

I still dislike it conceptually.  But let me do a technical review of
the latest version.

> And here is a new case from the LinuxCon Japan:
> 
> Developers from Hitach are now developing a new livedump mechanism for the
> same reason as ours. They have come to the situation *many times* that guest
> machines crashed due to host's failures, in particular, under development.

This has happened to me as well, possible even more times :).  I don't
use crash dumps for debugging but different people may use different
techniques.

> So they develop this mechanism to get crash dump while retaining the buggy
> situation between host and guest machine. The difference between theirs and
> ours is whether or not to use the feature on _customer's running machine_.


-- 
error compiling committee.c: too many arguments to function



  reply	other threads:[~2012-06-14 13:16 UTC|newest]

Thread overview: 69+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-05-16  7:50 [PATCH v2 0/5] Export offsets of VMCS fields as note information for kdump zhangyanfei
2012-05-16  7:50 ` zhangyanfei
2012-05-16  7:52 ` [PATCH v2 1/5] x86: Add helper variables and functions to hold VMCSINFO zhangyanfei
2012-05-16  7:52   ` zhangyanfei
2012-06-14 13:28   ` Avi Kivity
2012-06-14 13:28     ` Avi Kivity
2012-06-14 13:28     ` Avi Kivity
2012-05-16  7:54 ` [PATCH v2 2/5] KVM: Export symbols for module vmcsinfo-intel zhangyanfei
2012-05-16  7:54   ` zhangyanfei
2012-05-16  7:54   ` zhangyanfei
2012-05-16  7:55 ` [PATCH v2 3/5] KVM-INTEL: Add new module vmcsinfo-intel to fill VMCSINFO zhangyanfei
2012-05-16  7:55   ` zhangyanfei
2012-05-16  7:55   ` zhangyanfei
2012-06-14 13:37   ` Avi Kivity
2012-06-14 13:37     ` Avi Kivity
2012-06-14 13:37     ` Avi Kivity
2012-06-15  3:03     ` HATAYAMA Daisuke
2012-06-15  3:03       ` HATAYAMA Daisuke
2012-05-16  7:56 ` [PATCH v2 4/5] ksysfs: Export VMCSINFO via sysfs zhangyanfei
2012-05-16  7:56   ` zhangyanfei
2012-05-16  7:56   ` zhangyanfei
2012-05-16  7:57 ` [PATCH v2 5/5] Documentation: Add ABI entry for sysfs file vmcsinfo and vmcsinfo_maxsize zhangyanfei
2012-05-16  7:57   ` zhangyanfei
2012-05-16  7:57   ` zhangyanfei
2012-06-14 13:21   ` Avi Kivity
2012-06-14 13:21     ` Avi Kivity
2012-06-14 13:21     ` Avi Kivity
2012-05-20 17:43 ` [PATCH v2 0/5] Export offsets of VMCS fields as note information for kdump Avi Kivity
2012-05-20 17:43   ` Avi Kivity
2012-05-20 17:43   ` Avi Kivity
2012-05-21  2:32   ` Yanfei Zhang
2012-05-21  2:32     ` Yanfei Zhang
2012-05-21  8:34     ` Avi Kivity
2012-05-21  8:34       ` Avi Kivity
2012-05-21  8:34       ` Avi Kivity
2012-05-21  9:08       ` Yanfei Zhang
2012-05-21  9:08         ` Yanfei Zhang
2012-05-21  9:08         ` Yanfei Zhang
2012-05-21  9:36         ` Avi Kivity
2012-05-21  9:36           ` Avi Kivity
2012-05-21  9:36           ` Avi Kivity
2012-05-22  3:40           ` Yanfei Zhang
2012-05-22  3:40             ` Yanfei Zhang
2012-05-22  3:40             ` Yanfei Zhang
2012-05-28  5:25             ` Yanfei Zhang
2012-05-28  5:25               ` Yanfei Zhang
2012-05-28 13:28               ` Avi Kivity
2012-05-28 13:28                 ` Avi Kivity
2012-05-28 13:28                 ` Avi Kivity
2012-05-29  7:06                 ` Yanfei Zhang
2012-05-29  7:06                   ` Yanfei Zhang
2012-06-11  5:35                   ` Yanfei Zhang
2012-06-11  5:35                     ` Yanfei Zhang
2012-06-11  5:35                     ` Yanfei Zhang
2012-06-14 13:15                     ` Avi Kivity [this message]
2012-06-14 13:15                       ` Avi Kivity
2012-06-14 13:15                       ` Avi Kivity
2012-06-18  7:25                       ` YOSHIDA Masanori
2012-06-18  7:25                         ` YOSHIDA Masanori
2012-06-18  7:25                         ` YOSHIDA Masanori
2012-05-21 18:58 ` Eric Northup
2012-05-21 18:58   ` Eric Northup
2012-05-21 18:58   ` Eric Northup
2012-05-22  3:53   ` Yanfei Zhang
2012-05-22  3:53     ` Yanfei Zhang
2012-05-22  3:53     ` Yanfei Zhang
2012-05-22 20:53     ` Eric Northup
2012-05-22 20:53       ` Eric Northup
2012-05-22 20:53       ` Eric Northup

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=4FD9E3FF.4050906@redhat.com \
    --to=avi@redhat.com \
    --cc=dzickus@redhat.com \
    --cc=ebiederm@xmission.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=joerg.roedel@amd.com \
    --cc=kexec@lists.infradead.org \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=ludwig.nussel@suse.de \
    --cc=luto@mit.edu \
    --cc=masanori.yoshida.tv@hitachi.com \
    --cc=mtosatti@redhat.com \
    --cc=paul.gortmaker@windriver.com \
    --cc=zhangyanfei@cn.fujitsu.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.