RE: Help? Red Hat fails, Suse/Debian both work fine

RE: Help? Red Hat fails, Suse/Debian both work fine

[Xu, Anthony]

>It is likely some subtle difference (or bug), perhaps in mmap?

As I know, in Redhat, mmap can return NULL address, but seems xen
can't handle this situation, see below code segment:

In function vcpu_translate() of vcpu.c file

    else if (!region && warn_region0_address) {
        REGS *regs = vcpu_regs(vcpu);
        unsigned long viip = PSCB(vcpu,iip);
        unsigned long vipsr = PSCB(vcpu,ipsr);
        unsigned long iip = regs->cr_iip;
        unsigned long ipsr = regs->cr_ipsr;
        printk("vcpu_translate: bad address %p, viip=%p, vipsr=%p, iip=%p, ipsr=%p continuing\n", address, viip, vipsr, iip, ipsr);
    } 
warn_region0_address is turned off by default,
so maybe we can turn on warn_region0_address to see whether this is the root cause.

Thanks,
-Anthony 

>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Magenheimer, Dan
>(HP Labs Fort Collins)
>Sent: 2006年3月1日 5:45
>To: xen-devel
>Cc: yo.fujita; Akio Takebe; xen-ia64-devel@lists.xensource.com
>Subject: [Xen-devel] Help? Red Hat fails, Suse/Debian both work fine
>
>Hi all --
>
>We are seeing a strange problem where a recent cset causes
>Red Hat to fail domU boot on ia64 complaining of a hotplug
>problem but doesn't cause any problem for Suse or Debian.
>It is likely some subtle difference (or bug), perhaps in mmap?
>Suggestions/advice from anyone more familiar with distro
>differences (on ia64) would be appreciated.
>
>Changeset is xen-unstable 8783 ("Use /dev/kmem to map dom0
>xenstore page instead of abusing the foreign mapping interface.",
>Feb 8, committed by Christian).  Backing out this changeset
>or using the small patch below causes the problem to go away,
>so we have a workaround, but a root cause would be nice
>to know and fix.
>
>Full thread of discussion can be found here:
>http://lists.xensource.com/archives/html/xen-ia64-devel/2006-02/msg00104
>.html
>
>Thanks,
>Dan
>
>> -----Original Message-----
>> From: Akio Takebe [mailto:takebe_akio@jp.fujitsu.com]
>> Sent: Tuesday, February 28, 2006 2:47 AM
>> To: Magenheimer, Dan (HP Labs Fort Collins); yo.fujita;
>> xen-ia64-devel@lists.xensource.com
>> Cc: Akio Takebe
>> Subject: RE: [Xen-ia64-devel] Weekly benchmark results [2/3rd week]
>>
>> Hi, Dan
>>
>> I'm still debuging, but it is very difficult...
>> Much advice is welcome. :-)
>>
>> Now I can boot domU by using the following patch.
>>
>> diff -r 6c43118bdba8 tools/xenstore/xenstored_domain.c
>> --- a/tools/xenstore/xenstored_domain.c Fri Feb 24 15:41:08 2006 -0700
>> +++ b/tools/xenstore/xenstored_domain.c Tue Feb 28 18:20:16 2006 +0900
>> @@ -467,6 +467,7 @@ static int dom0_init(void)
>>         int rc, fd;
>>         evtchn_port_t port;
>>         unsigned long kva;
>> +       unsigned long mfn;
>>         char str[20];
>>         struct domain *dom0;
>>
>> @@ -500,9 +501,16 @@ static int dom0_init(void)
>>         if (fd == -1)
>>                 return -1;
>>
>> -       dom0->interface = mmap(NULL, getpagesize(),
>> PROT_READ|PROT_WRITE,
>> -                              MAP_SHARED, fd, kva);
>> -       if (dom0->interface == MAP_FAILED)
>> +       mfn=((0x0fffffffffffffff & kva) >>14);
>> +/*
>> +        dom0->interface = mmap(NULL, getpagesize(),
>> PROT_READ|PROT_WRITE,
>> +                               MAP_SHARED, fd, kva);
>> +*/
>> +       dom0->interface = xc_map_foreign_range(
>> +               *xc_handle, 0,
>> +               getpagesize(), PROT_READ|PROT_WRITE, mfn);
>> +       if (!dom0->interface)
>> +//     if (dom0->interface == MAP_FAILED)
>>                 goto outfd;
>>
>>         close(fd);
>>
>> Best Regards,
>>
>> Akio Takebe
>>
>> >Hi Akio --
>> >
>> >Any more progress on this issue?  If you are stuck,
>> >maybe we should post the problem to xen-devel to
>> >see if we can get help from a Red Hat person (since
>> >the problem doesn't occur on Suse or Debian).
>> >
>> >Thanks,
>> >Dan
>> >
>> >> -----Original Message-----
>> >> From: Akio Takebe [mailto:takebe_akio@jp.fujitsu.com]
>> >> Sent: Thursday, February 23, 2006 8:45 PM
>> >> To: Magenheimer, Dan (HP Labs Fort Collins); yo.fujita;
>> >> xen-ia64-devel@lists.xensource.com
>> >> Cc: Akio Takebe
>> >> Subject: RE: [Xen-ia64-devel] Weekly benchmark results [2/3rd week]
>> >>
>> >> Hi, Dan and Alex
>> >>
>> >> I think this issue is only on ia64.
>> >> I seem that kmem_map@drivers/char/mem.c is used on ia64,
>> >> but mem_map@drivers/xen/char/mem.c is used on x86.
>> >> So I think pfn or kva aren't set correctly.
>> >> We tried to boot domU with revesing cset xen-ia64-ustable.8790
>> >> and it was good work.
>> >>
>> >> I'm still debugging it. :-<
>> >>
>> >> Best Regards,
>> >>
>> >> Akio Takebe
>> >>
>> >> >Confirmed cset xen-unstable 8783 fails while 8782 succeeds.
>> >> >
>> >> >Perhaps there's something different about mmap on RH
>> >> >vs Suse and Debian?  Perhaps only on ia64?
>> >> >
>> >>
>> >>
>> >>
>>
>>
>>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel

Re: [Xen-devel] Help? Red Hat fails, Suse/Debian both work fine

[Tristan Gingold]

Le Mercredi 01 Mars 2006 02:30, Xu, Anthony a écrit :
> >It is likely some subtle difference (or bug), perhaps in mmap?
Hi all,

yesterday I got the same bug as Akio.  Today it is working again.  I have just 
reinstalled tools+xen+kernel.
Akio, could you try to reinstall xen+tools+kernel using a clean repository ?
(make clean may be not enough).

It seems *really* strange that only a few of us got the bug because it seems 
to be related only to Xen+Xenstore+Linux.  Therefore it should not depend on 
the distribution.

Tristan.

RE: Help? Red Hat fails, Suse/Debian both work fine

[Akio Takebe]

Hi, Anthony

Thank you for your advice.
I checked retun value of mmap(), and it is not NULL.
I'll check vcpu_translate().

Best Regards, 

Akio Takebe

>>It is likely some subtle difference (or bug), perhaps in mmap?
>
>As I know, in Redhat, mmap can return NULL address, but seems xen
>can't handle this situation, see below code segment:
>
>In function vcpu_translate() of vcpu.c file
>
>    else if (!region && warn_region0_address) {
>        REGS *regs = vcpu_regs(vcpu);
>        unsigned long viip = PSCB(vcpu,iip);
>        unsigned long vipsr = PSCB(vcpu,ipsr);
>        unsigned long iip = regs->cr_iip;
>        unsigned long ipsr = regs->cr_ipsr;
>        printk("vcpu_translate: bad address %p, viip=%p, vipsr=%p, iip=%p, 
>ipsr=%p continuing\n", address, viip, vipsr, iip, ipsr);
>    } 
>warn_region0_address is turned off by default,
>so maybe we can turn on warn_region0_address to see whether this is the 
>root cause.
>
>Thanks,
>-Anthony 
>
>>-----Original Message-----
>>From: xen-devel-bounces@lists.xensource.com
>>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Magenheimer, Dan
>>(HP Labs Fort Collins)
>>Sent: 2006ト\xF3\x14ヤツ1ネユ 5:45
>>To: xen-devel
>>Cc: yo.fujita; Akio Takebe; xen-ia64-devel@lists.xensource.com
>>Subject: [Xen-devel] Help? Red Hat fails, Suse/Debian both work fine
>>
>>Hi all --
>>
>>We are seeing a strange problem where a recent cset causes
>>Red Hat to fail domU boot on ia64 complaining of a hotplug
>>problem but doesn't cause any problem for Suse or Debian.
>>It is likely some subtle difference (or bug), perhaps in mmap?
>>Suggestions/advice from anyone more familiar with distro
>>differences (on ia64) would be appreciated.
>>
>>Changeset is xen-unstable 8783 ("Use /dev/kmem to map dom0
>>xenstore page instead of abusing the foreign mapping interface.",
>>Feb 8, committed by Christian).  Backing out this changeset
>>or using the small patch below causes the problem to go away,
>>so we have a workaround, but a root cause would be nice
>>to know and fix.
>>
>>Full thread of discussion can be found here:
>>http://lists.xensource.com/archives/html/xen-ia64-devel/2006-02/msg00104
>>.html
>>
>>Thanks,
>>Dan
>>
>>> -----Original Message-----
>>> From: Akio Takebe [mailto:takebe_akio@jp.fujitsu.com]
>>> Sent: Tuesday, February 28, 2006 2:47 AM
>>> To: Magenheimer, Dan (HP Labs Fort Collins); yo.fujita;
>>> xen-ia64-devel@lists.xensource.com
>>> Cc: Akio Takebe
>>> Subject: RE: [Xen-ia64-devel] Weekly benchmark results [2/3rd week]
>>>
>>> Hi, Dan
>>>
>>> I'm still debuging, but it is very difficult...
>>> Much advice is welcome. :-)
>>>
>>> Now I can boot domU by using the following patch.
>>>
>>> diff -r 6c43118bdba8 tools/xenstore/xenstored_domain.c
>>> --- a/tools/xenstore/xenstored_domain.c Fri Feb 24 15:41:08 2006 -0700
>>> +++ b/tools/xenstore/xenstored_domain.c Tue Feb 28 18:20:16 2006 +0900
>>> @@ -467,6 +467,7 @@ static int dom0_init(void)
>>>         int rc, fd;
>>>         evtchn_port_t port;
>>>         unsigned long kva;
>>> +       unsigned long mfn;
>>>         char str[20];
>>>         struct domain *dom0;
>>>
>>> @@ -500,9 +501,16 @@ static int dom0_init(void)
>>>         if (fd == -1)
>>>                 return -1;
>>>
>>> -       dom0->interface = mmap(NULL, getpagesize(),
>>> PROT_READ|PROT_WRITE,
>>> -                              MAP_SHARED, fd, kva);
>>> -       if (dom0->interface == MAP_FAILED)
>>> +       mfn=((0x0fffffffffffffff & kva) >>14);
>>> +/*
>>> +        dom0->interface = mmap(NULL, getpagesize(),
>>> PROT_READ|PROT_WRITE,
>>> +                               MAP_SHARED, fd, kva);
>>> +*/
>>> +       dom0->interface = xc_map_foreign_range(
>>> +               *xc_handle, 0,
>>> +               getpagesize(), PROT_READ|PROT_WRITE, mfn);
>>> +       if (!dom0->interface)
>>> +//     if (dom0->interface == MAP_FAILED)
>>>                 goto outfd;
>>>
>>>         close(fd);
>>>
>>> Best Regards,
>>>
>>> Akio Takebe
>>>
>>> >Hi Akio --
>>> >
>>> >Any more progress on this issue?  If you are stuck,
>>> >maybe we should post the problem to xen-devel to
>>> >see if we can get help from a Red Hat person (since
>>> >the problem doesn't occur on Suse or Debian).
>>> >
>>> >Thanks,
>>> >Dan
>>> >
>>> >> -----Original Message-----
>>> >> From: Akio Takebe [mailto:takebe_akio@jp.fujitsu.com]
>>> >> Sent: Thursday, February 23, 2006 8:45 PM
>>> >> To: Magenheimer, Dan (HP Labs Fort Collins); yo.fujita;
>>> >> xen-ia64-devel@lists.xensource.com
>>> >> Cc: Akio Takebe
>>> >> Subject: RE: [Xen-ia64-devel] Weekly benchmark results [2/3rd week]
>>> >>
>>> >> Hi, Dan and Alex
>>> >>
>>> >> I think this issue is only on ia64.
>>> >> I seem that kmem_map@drivers/char/mem.c is used on ia64,
>>> >> but mem_map@drivers/xen/char/mem.c is used on x86.
>>> >> So I think pfn or kva aren't set correctly.
>>> >> We tried to boot domU with revesing cset xen-ia64-ustable.8790
>>> >> and it was good work.
>>> >>
>>> >> I'm still debugging it. :-<
>>> >>
>>> >> Best Regards,
>>> >>
>>> >> Akio Takebe
>>> >>
>>> >> >Confirmed cset xen-unstable 8783 fails while 8782 succeeds.
>>> >> >
>>> >> >Perhaps there's something different about mmap on RH
>>> >> >vs Suse and Debian?  Perhaps only on ia64?
>>> >> >
>>> >>
>>> >>
>>> >>
>>>
>>>
>>>
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel

Re: Help? Red Hat fails, Suse/Debian both work fine

[Akio Takebe]

Hi, Tristan

Thank you. I'll try to reinstall soon.
If reinstall is a root cause, it is mean some tools are not overwrite 
when we run "make install-tools".

Best Regards,

Akio Takebe

>Le Mercredi 01 Mars 2006 02:30, Xu, Anthony a ィヲcrit :
>> >It is likely some subtle difference (or bug), perhaps in mmap?
>Hi all,
>
>yesterday I got the same bug as Akio.  Today it is working again.  I have 
>just 
>reinstalled tools+xen+kernel.
>Akio, could you try to reinstall xen+tools+kernel using a clean repository ?
>(make clean may be not enough).
>
>It seems *really* strange that only a few of us got the bug because it seems 
>to be related only to Xen+Xenstore+Linux.  Therefore it should not depend on 
>the distribution.
>
>Tristan.
>

Re: [Xen-devel] Help? Red Hat fails, Suse/Debian both work fine

[Keir Fraser]

On 2 Mar 2006, at 05:43, Tian, Kevin wrote:

> 	However the interesting thing is, following Cset is only for
> changing way to map dom0's store page, instead of domU. DomU's
> store page is still mapped by foreign page map. If above hint is real
> cause, xend start can fail earlier due to incorrect mapping when
> introducing dom0 into xenstore. However you all observe the bug
> bothering only when domU is booting...

The dom0 connection is only used in anger when first creating a domU, 
to set up device backend state. Until that time it could be screwed and 
you probably would not notice.

  -- Keir

RE: [Xen-devel] Help? Red Hat fails, Suse/Debian both work fine

[Tian, Kevin]

>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>Sent: 2006年3月2日 16:26
>
>On 2 Mar 2006, at 05:43, Tian, Kevin wrote:
>
>> 	However the interesting thing is, following Cset is only for
>> changing way to map dom0's store page, instead of domU. DomU's
>> store page is still mapped by foreign page map. If above hint is real
>> cause, xend start can fail earlier due to incorrect mapping when
>> introducing dom0 into xenstore. However you all observe the bug
>> bothering only when domU is booting...
>
>The dom0 connection is only used in anger when first creating a domU,
>to set up device backend state. Until that time it could be screwed and
>you probably would not notice.
>
>  -- Keir

Thanks. Make sense.
-Kevin

Re: [Xen-devel] Help? Red Hat fails, Suse/Debian both work fine

[Akio Takebe]

Hi, Tristan

I tried reinstall and I have the same error.
But you are right. 
"make install-tools" didn't change under /usr/lib/python/xen.
This may be another issue, but this is important issue.
We'll check Makefile after this issue is fixed.

Best Regards,

Akio Takebe

>Hi, Tristan
>
>Thank you. I'll try to reinstall soon.
>If reinstall is a root cause, it is mean some tools are not overwrite 
>when we run "make install-tools".
>
>Best Regards,
>
>Akio Takebe
>
>>Le Mercredi 01 Mars 2006 02:30, Xu, Anthony a ィヲcrit :
>>> >It is likely some subtle difference (or bug), perhaps in mmap?
>>Hi all,
>>
>>yesterday I got the same bug as Akio.  Today it is working again.  I have 
>>just 
>>reinstalled tools+xen+kernel.
>>Akio, could you try to reinstall xen+tools+kernel using a clean repository ?
>>(make clean may be not enough).
>>
>>It seems *really* strange that only a few of us got the bug because it 
>>seems 
>>to be related only to Xen+Xenstore+Linux.  Therefore it should not depend 
>>on 
>>the distribution.
>>
>>Tristan.
>>
>