why xenstore-* hangs if xenstored is not started

why xenstore-* hangs if xenstored is not started

[Yu Zhiguo]

 Running xenstore-* when xenstored is not started, it will hang and
cannot accept any signal, e.g. SIGINT.
 I seems that the process is blocked at accessing '/proc/xen/xenbus'.

 # mount -t xenfs xenfs /proc/xen/
 # xenstore-ls

 call trace:
 do_ls() -> xs_directory() -> xs_talkv() -> xs_write_all() -> write()

 BLOCK is no problem but why process cannot accept signal, Any opinion?


 If start xenstored in other console, xenstore-ls executes and then
receive the signal, seems the signal is pended...

-- 
Regards
Yu Zhiguo

Re: why xenstore-* hangs if xenstored is not started

[Vincent Hanquez]

On 22/06/10 08:32, Yu Zhiguo wrote:
>   Running xenstore-* when xenstored is not started, it will hang and
> cannot accept any signal, e.g. SIGINT.
>   I seems that the process is blocked at accessing '/proc/xen/xenbus'.
>
>   # mount -t xenfs xenfs /proc/xen/
>   # xenstore-ls
>
>   call trace:
>   do_ls() ->  xs_directory() ->  xs_talkv() ->  xs_write_all() ->  write()
>
>   BLOCK is no problem but why process cannot accept signal, Any opinion?
>
>
>   If start xenstored in other console, xenstore-ls executes and then
> receive the signal, seems the signal is pended...
>

That's the common behavior of task stuck in a kernel operation (ps aux 
will show the task at D, and /proc/<pid>/wchan will tell you exactly 
where it's blocked in the kernel).

the write is going to wait for a xenstored reply in kernel mode since by 
default the xenstore-* tools use the kernel xenstore page to communicate 
with xenstored.

The unix socket operations doesn't have this "problems" (xenstore-* -s)

-- 
Vincent

Re: why xenstore-* hangs if xenstored is not started

[Keir Fraser]

On 22/06/2010 08:32, "Yu Zhiguo" <yuzg@cn.fujitsu.com> wrote:

>  Running xenstore-* when xenstored is not started, it will hang and
> cannot accept any signal, e.g. SIGINT.
>  I seems that the process is blocked at accessing '/proc/xen/xenbus'.
> 
>  # mount -t xenfs xenfs /proc/xen/
>  # xenstore-ls
> 
>  call trace:
>  do_ls() -> xs_directory() -> xs_talkv() -> xs_write_all() -> write()
> 
>  BLOCK is no problem but why process cannot accept signal, Any opinion?

Does the xenstore driver in the dom0 kernel put the process into
uninterruptible sleep? The 2.6.18 kernel simply returns -ENODEV in this case
I think. So perhaps pv_ops has different and inferior behaviour. At least
using interruptible sleep would be preferable.

 -- Keir

>  If start xenstored in other console, xenstore-ls executes and then
> receive the signal, seems the signal is pended...

Re: why xenstore-* hangs if xenstored is not started

[Vincent Hanquez]

On 22/06/10 08:45, Keir Fraser wrote:
> Does the xenstore driver in the dom0 kernel put the process into
> uninterruptible sleep? The 2.6.18 kernel simply returns -ENODEV in this case
> I think. So perhaps pv_ops has different and inferior behaviour.

the behavior was the same in 2.6.18.

-- 
Vincent

Re: why xenstore-* hangs if xenstored is not started

[Yu Zhiguo]

Hi,

Vincent Hanquez wrote:

> That's the common behavior of task stuck in a kernel operation (ps aux
> will show the task at D, and /proc/<pid>/wchan will tell you exactly
> where it's blocked in the kernel).
> 
> the write is going to wait for a xenstored reply in kernel mode since by
> default the xenstore-* tools use the kernel xenstore page to communicate
> with xenstored.
> 
> The unix socket operations doesn't have this "problems" (xenstore-* -s)
> 

Thanks~~
I confirmed the task is at D+ and there are no this "problem" if use option -s.
But I think S+ is better than D+...


Yu

Re: why xenstore-* hangs if xenstored is not started

[Keir Fraser]

On 22/06/2010 08:56, "Vincent Hanquez" <Vincent.Hanquez@eu.citrix.com>
wrote:

> On 22/06/10 08:45, Keir Fraser wrote:
>> Does the xenstore driver in the dom0 kernel put the process into
>> uninterruptible sleep? The 2.6.18 kernel simply returns -ENODEV in this case
>> I think. So perhaps pv_ops has different and inferior behaviour.
> 
> the behavior was the same in 2.6.18.

Ah, I just see 'if (!is_xenstored_ready()) return -ENODEV' in 2.6.18's
xenbus_dev.c. I assumed that would have an effect on this case.

 -- Keir

Re: why xenstore-* hangs if xenstored is not started

[Yu Zhiguo]

Hi Keir,

Keir Fraser wrote:
>>
>>  BLOCK is no problem but why process cannot accept signal, Any opinion?
> 
> Does the xenstore driver in the dom0 kernel put the process into
> uninterruptible sleep? The 2.6.18 kernel simply returns -ENODEV in this case
> I think. So perhaps pv_ops has different and inferior behaviour. At least
> using interruptible sleep would be preferable.
> 

Yes, the process is uninterruptible.
I agree with you, interruptible sleep is user-friendliness.

Yu

Re: why xenstore-* hangs if xenstored is not started

[Vincent Hanquez]

On 22/06/10 09:06, Keir Fraser wrote:
> On 22/06/2010 08:56, "Vincent Hanquez"<Vincent.Hanquez@eu.citrix.com>
> wrote:
>
>> On 22/06/10 08:45, Keir Fraser wrote:
>>> Does the xenstore driver in the dom0 kernel put the process into
>>> uninterruptible sleep? The 2.6.18 kernel simply returns -ENODEV in this case
>>> I think. So perhaps pv_ops has different and inferior behaviour.
>>
>> the behavior was the same in 2.6.18.
>
> Ah, I just see 'if (!is_xenstored_ready()) return -ENODEV' in 2.6.18's
> xenbus_dev.c. I assumed that would have an effect on this case.

hm, maybe it does afterall, and I'm confuse this with a more general 
case; the D state would happens if xenstored doesn't want to reply or is 
dead. This behavior has been there as long as I can remember.

I'm pretty sure though that a while ago (relatively old XenServer 
kernel) starting a xenstore-* without starting xenstored would make it 
stuck until xenstored start.

-- 
Vincent

Re: why xenstore-* hangs if xenstored is not started

[Vincent Hanquez]

On 22/06/10 08:58, Yu Zhiguo wrote:
> Hi,
>
> Vincent Hanquez wrote:
>
>> That's the common behavior of task stuck in a kernel operation (ps aux
>> will show the task at D, and /proc/<pid>/wchan will tell you exactly
>> where it's blocked in the kernel).
>>
>> the write is going to wait for a xenstored reply in kernel mode since by
>> default the xenstore-* tools use the kernel xenstore page to communicate
>> with xenstored.
>>
>> The unix socket operations doesn't have this "problems" (xenstore-* -s)
>>
>
> Thanks~~
> I confirmed the task is at D+ and there are no this "problem" if use option -s.
> But I think S+ is better than D+...

I don't think it's very easy to have interruptible sleep here, it would 
change radically the protocol. the kernel xenstored page need to be 
handled "atomically" since requests and their replied cannot be 
intertwined. Maybe a timeout would be the one requiring the less change ?

-- 
Vincent

Re: why xenstore-* hangs if xenstored is not started

[Yu Zhiguo]

Hi Vincent,

Vincent Hanquez wrote:
>>
>> Thanks~~
>> I confirmed the task is at D+ and there are no this "problem" if use
>> option -s.
>> But I think S+ is better than D+...
> 
> I don't think it's very easy to have interruptible sleep here, it would
> change radically the protocol. the kernel xenstored page need to be
> handled "atomically" since requests and their replied cannot be
> intertwined. Maybe a timeout would be the one requiring the less change ?
> 

 I'm not familiar with those code and not sure how can fix it,
but I think a timeout is better than doing nothing.


-- 
Regards
Yu Zhiguo