* RE: Help? Red Hat fails, Suse/Debian both work fine
@ 2006-03-02 5:43 Tian, Kevin
2006-03-02 6:04 ` [Xen-ia64-devel] " Yoshi.Oguchi
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Tian, Kevin @ 2006-03-02 5:43 UTC (permalink / raw)
To: Akio Takebe, Xu, Anthony, Magenheimer, Dan (HP Labs Fort Collins),
xen-devel
Cc: yo.fujita, xen-ia64-devel
Hi, Akio,
Currently linux-2.6-xen-sparse/driver/xen/char/ is not included
in compilation for xen/ia64, so you're still using linux-style kmem
path.
Try to compile that directory into your xenlinux image, and
define ARCH_HAS_DEV_MEM, and then see anything changed
for you.
However the interesting thing is, following Cset is only for
changing way to map dom0's store page, instead of domU. DomU's
store page is still mapped by foreign page map. If above hint is real
cause, xend start can fail earlier due to incorrect mapping when
introducing dom0 into xenstore. However you all observe the bug
bothering only when domU is booting...
Thanks,
Kevin
>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Akio Takebe
>Sent: 2006年3月2日 8:44
>To: Xu, Anthony; Magenheimer, Dan (HP Labs Fort Collins); xen-devel
>Cc: yo.fujita; Akio Takebe; xen-ia64-devel@lists.xensource.com
>Subject: RE: [Xen-devel] Help? Red Hat fails, Suse/Debian both work fine
>
>Hi, Anthony
>
>Thank you for your advice.
>I checked retun value of mmap(), and it is not NULL.
>I'll check vcpu_translate().
>
>Best Regards,
>
>Akio Takebe
>
>>>It is likely some subtle difference (or bug), perhaps in mmap?
>>
>>As I know, in Redhat, mmap can return NULL address, but seems xen
>>can't handle this situation, see below code segment:
>>
>>In function vcpu_translate() of vcpu.c file
>>
>> else if (!region && warn_region0_address) {
>> REGS *regs = vcpu_regs(vcpu);
>> unsigned long viip = PSCB(vcpu,iip);
>> unsigned long vipsr = PSCB(vcpu,ipsr);
>> unsigned long iip = regs->cr_iip;
>> unsigned long ipsr = regs->cr_ipsr;
>> printk("vcpu_translate: bad address %p, viip=%p, vipsr=%p, iip=%p,
>>ipsr=%p continuing\n", address, viip, vipsr, iip, ipsr);
>> }
>>warn_region0_address is turned off by default,
>>so maybe we can turn on warn_region0_address to see whether this is the
>>root cause.
>>
>>Thanks,
>>-Anthony
>>
>>>-----Original Message-----
>>>From: xen-devel-bounces@lists.xensource.com
>>>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of
>Magenheimer, Dan
>>>(HP Labs Fort Collins)
>>>Sent: 2006ト・ヤツ1ネユ 5:45
>>>To: xen-devel
>>>Cc: yo.fujita; Akio Takebe; xen-ia64-devel@lists.xensource.com
>>>Subject: [Xen-devel] Help? Red Hat fails, Suse/Debian both work fine
>>>
>>>Hi all --
>>>
>>>We are seeing a strange problem where a recent cset causes
>>>Red Hat to fail domU boot on ia64 complaining of a hotplug
>>>problem but doesn't cause any problem for Suse or Debian.
>>>It is likely some subtle difference (or bug), perhaps in mmap?
>>>Suggestions/advice from anyone more familiar with distro
>>>differences (on ia64) would be appreciated.
>>>
>>>Changeset is xen-unstable 8783 ("Use /dev/kmem to map dom0
>>>xenstore page instead of abusing the foreign mapping interface.",
>>>Feb 8, committed by Christian). Backing out this changeset
>>>or using the small patch below causes the problem to go away,
>>>so we have a workaround, but a root cause would be nice
>>>to know and fix.
>>>
>>>Full thread of discussion can be found here:
>>>http://lists.xensource.com/archives/html/xen-ia64-devel/2006-02/msg00104
>>>.html
>>>
>>>Thanks,
>>>Dan
>>>
>>>> -----Original Message-----
>>>> From: Akio Takebe [mailto:takebe_akio@jp.fujitsu.com]
>>>> Sent: Tuesday, February 28, 2006 2:47 AM
>>>> To: Magenheimer, Dan (HP Labs Fort Collins); yo.fujita;
>>>> xen-ia64-devel@lists.xensource.com
>>>> Cc: Akio Takebe
>>>> Subject: RE: [Xen-ia64-devel] Weekly benchmark results [2/3rd week]
>>>>
>>>> Hi, Dan
>>>>
>>>> I'm still debuging, but it is very difficult...
>>>> Much advice is welcome. :-)
>>>>
>>>> Now I can boot domU by using the following patch.
>>>>
>>>> diff -r 6c43118bdba8 tools/xenstore/xenstored_domain.c
>>>> --- a/tools/xenstore/xenstored_domain.c Fri Feb 24 15:41:08 2006 -0700
>>>> +++ b/tools/xenstore/xenstored_domain.c Tue Feb 28 18:20:16 2006
>+0900
>>>> @@ -467,6 +467,7 @@ static int dom0_init(void)
>>>> int rc, fd;
>>>> evtchn_port_t port;
>>>> unsigned long kva;
>>>> + unsigned long mfn;
>>>> char str[20];
>>>> struct domain *dom0;
>>>>
>>>> @@ -500,9 +501,16 @@ static int dom0_init(void)
>>>> if (fd == -1)
>>>> return -1;
>>>>
>>>> - dom0->interface = mmap(NULL, getpagesize(),
>>>> PROT_READ|PROT_WRITE,
>>>> - MAP_SHARED, fd, kva);
>>>> - if (dom0->interface == MAP_FAILED)
>>>> + mfn=((0x0fffffffffffffff & kva) >>14);
>>>> +/*
>>>> + dom0->interface = mmap(NULL, getpagesize(),
>>>> PROT_READ|PROT_WRITE,
>>>> + MAP_SHARED, fd, kva);
>>>> +*/
>>>> + dom0->interface = xc_map_foreign_range(
>>>> + *xc_handle, 0,
>>>> + getpagesize(), PROT_READ|PROT_WRITE, mfn);
>>>> + if (!dom0->interface)
>>>> +// if (dom0->interface == MAP_FAILED)
>>>> goto outfd;
>>>>
>>>> close(fd);
>>>>
>>>> Best Regards,
>>>>
>>>> Akio Takebe
>>>>
>>>> >Hi Akio --
>>>> >
>>>> >Any more progress on this issue? If you are stuck,
>>>> >maybe we should post the problem to xen-devel to
>>>> >see if we can get help from a Red Hat person (since
>>>> >the problem doesn't occur on Suse or Debian).
>>>> >
>>>> >Thanks,
>>>> >Dan
>>>> >
>>>> >> -----Original Message-----
>>>> >> From: Akio Takebe [mailto:takebe_akio@jp.fujitsu.com]
>>>> >> Sent: Thursday, February 23, 2006 8:45 PM
>>>> >> To: Magenheimer, Dan (HP Labs Fort Collins); yo.fujita;
>>>> >> xen-ia64-devel@lists.xensource.com
>>>> >> Cc: Akio Takebe
>>>> >> Subject: RE: [Xen-ia64-devel] Weekly benchmark results [2/3rd week]
>>>> >>
>>>> >> Hi, Dan and Alex
>>>> >>
>>>> >> I think this issue is only on ia64.
>>>> >> I seem that kmem_map@drivers/char/mem.c is used on ia64,
>>>> >> but mem_map@drivers/xen/char/mem.c is used on x86.
>>>> >> So I think pfn or kva aren't set correctly.
>>>> >> We tried to boot domU with revesing cset xen-ia64-ustable.8790
>>>> >> and it was good work.
>>>> >>
>>>> >> I'm still debugging it. :-<
>>>> >>
>>>> >> Best Regards,
>>>> >>
>>>> >> Akio Takebe
>>>> >>
>>>> >> >Confirmed cset xen-unstable 8783 fails while 8782 succeeds.
>>>> >> >
>>>> >> >Perhaps there's something different about mmap on RH
>>>> >> >vs Suse and Debian? Perhaps only on ia64?
>>>> >> >
>>>> >>
>>>> >>
>>>> >>
>>>>
>>>>
>>>>
>>>
>>>_______________________________________________
>>>Xen-devel mailing list
>>>Xen-devel@lists.xensource.com
>>>http://lists.xensource.com/xen-devel
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Xen-ia64-devel] RE: Help? Red Hat fails, Suse/Debian both work fine
2006-03-02 5:43 Help? Red Hat fails, Suse/Debian both work fine Tian, Kevin
@ 2006-03-02 6:04 ` Yoshi.Oguchi
2006-03-02 8:26 ` [Xen-devel] " Keir Fraser
2006-03-02 15:18 ` Akio Takebe
2 siblings, 0 replies; 8+ messages in thread
From: Yoshi.Oguchi @ 2006-03-02 6:04 UTC (permalink / raw)
To: Tian, Kevin
Cc: Magenheimer, Dan (HP Labs Fort Collins), Xu, Anthony, Akio Takebe,
xen-devel, xen-ia64-devel
Hi Kevin,
>page is still mapped by foreign page map. If above hint is real
>cause, xend start can fail earlier due to incorrect mapping when
>introducing dom0 into xenstore. However you all observe the bug
>bothering only when domU is booting...
>
As you suspect, we observed xenstored behaves incorrectly
from the start.
After booting Dom0, typing "xenstore-list /" results in
"bad_client" error (or just hangs).
Apparently, xenstored can't read mapped ring buffer.
Akio is now debugging on it, also he'll try to see if
anything changes when /driver/xen/char is included.
Thanks,
Yoshi Oguchi
--
Tian, Kevin wrote:
>Hi, Akio,
> Currently linux-2.6-xen-sparse/driver/xen/char/ is not included
>in compilation for xen/ia64, so you're still using linux-style kmem
>path.
>
> Try to compile that directory into your xenlinux image, and
>define ARCH_HAS_DEV_MEM, and then see anything changed
>for you.
>
> However the interesting thing is, following Cset is only for
>changing way to map dom0's store page, instead of domU. DomU's
>store page is still mapped by foreign page map. If above hint is real
>cause, xend start can fail earlier due to incorrect mapping when
>introducing dom0 into xenstore. However you all observe the bug
>bothering only when domU is booting...
>
>Thanks,
>Kevin
>
>>-----Original Message-----
>>From: xen-devel-bounces@lists.xensource.com
>>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Akio Takebe
>>Sent: 2006年3月2日 8:44
>>To: Xu, Anthony; Magenheimer, Dan (HP Labs Fort Collins); xen-devel
>>Cc: yo.fujita; Akio Takebe; xen-ia64-devel@lists.xensource.com
>>Subject: RE: [Xen-devel] Help? Red Hat fails, Suse/Debian both work fine
>>
>>Hi, Anthony
>>
>>Thank you for your advice.
>>I checked retun value of mmap(), and it is not NULL.
>>I'll check vcpu_translate().
>>
>>Best Regards,
>>
>>Akio Takebe
>>
>>>>It is likely some subtle difference (or bug), perhaps in mmap?
>>>
>>>As I know, in Redhat, mmap can return NULL address, but seems xen
>>>can't handle this situation, see below code segment:
>>>
>>>In function vcpu_translate() of vcpu.c file
>>>
>>> else if (!region && warn_region0_address) {
>>> REGS *regs = vcpu_regs(vcpu);
>>> unsigned long viip = PSCB(vcpu,iip);
>>> unsigned long vipsr = PSCB(vcpu,ipsr);
>>> unsigned long iip = regs->cr_iip;
>>> unsigned long ipsr = regs->cr_ipsr;
>>> printk("vcpu_translate: bad address %p, viip=%p, vipsr=%p, iip=%
p,
>>>ipsr=%p continuing\n", address, viip, vipsr, iip, ipsr);
>>> }
>>>warn_region0_address is turned off by default,
>>>so maybe we can turn on warn_region0_address to see whether this is the
>>>root cause.
>>>
>>>Thanks,
>>>-Anthony
>>>
>>>>-----Original Message-----
>>>>From: xen-devel-bounces@lists.xensource.com
>>>>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of
>>Magenheimer, Dan
>>>>(HP Labs Fort Collins)
>>>>Sent: 2006ト・ヤツ1ネユ 5:45
>>>>To: xen-devel
>>>>Cc: yo.fujita; Akio Takebe; xen-ia64-devel@lists.xensource.com
>>>>Subject: [Xen-devel] Help? Red Hat fails, Suse/Debian both work fine
>>>>
>>>>Hi all --
>>>>
>>>>We are seeing a strange problem where a recent cset causes
>>>>Red Hat to fail domU boot on ia64 complaining of a hotplug
>>>>problem but doesn't cause any problem for Suse or Debian.
>>>>It is likely some subtle difference (or bug), perhaps in mmap?
>>>>Suggestions/advice from anyone more familiar with distro
>>>>differences (on ia64) would be appreciated.
>>>>
>>>>Changeset is xen-unstable 8783 ("Use /dev/kmem to map dom0
>>>>xenstore page instead of abusing the foreign mapping interface.",
>>>>Feb 8, committed by Christian). Backing out this changeset
>>>>or using the small patch below causes the problem to go away,
>>>>so we have a workaround, but a root cause would be nice
>>>>to know and fix.
>>>>
>>>>Full thread of discussion can be found here:
>>>>http://lists.xensource.com/archives/html/xen-ia64-devel/2006-02/msg00104
>>>>.html
>>>>
>>>>Thanks,
>>>>Dan
>>>>
>>>>> -----Original Message-----
>>>>> From: Akio Takebe [mailto:takebe_akio@jp.fujitsu.com]
>>>>> Sent: Tuesday, February 28, 2006 2:47 AM
>>>>> To: Magenheimer, Dan (HP Labs Fort Collins); yo.fujita;
>>>>> xen-ia64-devel@lists.xensource.com
>>>>> Cc: Akio Takebe
>>>>> Subject: RE: [Xen-ia64-devel] Weekly benchmark results [2/3rd week]
>>>>>
>>>>> Hi, Dan
>>>>>
>>>>> I'm still debuging, but it is very difficult...
>>>>> Much advice is welcome. :-)
>>>>>
>>>>> Now I can boot domU by using the following patch.
>>>>>
>>>>> diff -r 6c43118bdba8 tools/xenstore/xenstored_domain.c
>>>>> --- a/tools/xenstore/xenstored_domain.c Fri Feb 24 15:41:08 2006 -
0700
>>>>> +++ b/tools/xenstore/xenstored_domain.c Tue Feb 28 18:20:16 2006
>>+0900
>>>>> @@ -467,6 +467,7 @@ static int dom0_init(void)
>>>>> int rc, fd;
>>>>> evtchn_port_t port;
>>>>> unsigned long kva;
>>>>> + unsigned long mfn;
>>>>> char str[20];
>>>>> struct domain *dom0;
>>>>>
>>>>> @@ -500,9 +501,16 @@ static int dom0_init(void)
>>>>> if (fd == -1)
>>>>> return -1;
>>>>>
>>>>> - dom0->interface = mmap(NULL, getpagesize(),
>>>>> PROT_READ|PROT_WRITE,
>>>>> - MAP_SHARED, fd, kva);
>>>>> - if (dom0->interface == MAP_FAILED)
>>>>> + mfn=((0x0fffffffffffffff & kva) >>14);
>>>>> +/*
>>>>> + dom0->interface = mmap(NULL, getpagesize(),
>>>>> PROT_READ|PROT_WRITE,
>>>>> + MAP_SHARED, fd, kva);
>>>>> +*/
>>>>> + dom0->interface = xc_map_foreign_range(
>>>>> + *xc_handle, 0,
>>>>> + getpagesize(), PROT_READ|PROT_WRITE, mfn);
>>>>> + if (!dom0->interface)
>>>>> +// if (dom0->interface == MAP_FAILED)
>>>>> goto outfd;
>>>>>
>>>>> close(fd);
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Akio Takebe
>>>>>
>>>>> >Hi Akio --
>>>>> >
>>>>> >Any more progress on this issue? If you are stuck,
>>>>> >maybe we should post the problem to xen-devel to
>>>>> >see if we can get help from a Red Hat person (since
>>>>> >the problem doesn't occur on Suse or Debian).
>>>>> >
>>>>> >Thanks,
>>>>> >Dan
>>>>> >
>>>>> >> -----Original Message-----
>>>>> >> From: Akio Takebe [mailto:takebe_akio@jp.fujitsu.com]
>>>>> >> Sent: Thursday, February 23, 2006 8:45 PM
>>>>> >> To: Magenheimer, Dan (HP Labs Fort Collins); yo.fujita;
>>>>> >> xen-ia64-devel@lists.xensource.com
>>>>> >> Cc: Akio Takebe
>>>>> >> Subject: RE: [Xen-ia64-devel] Weekly benchmark results [2/3rd
week]
>>>>> >>
>>>>> >> Hi, Dan and Alex
>>>>> >>
>>>>> >> I think this issue is only on ia64.
>>>>> >> I seem that kmem_map@drivers/char/mem.c is used on ia64,
>>>>> >> but mem_map@drivers/xen/char/mem.c is used on x86.
>>>>> >> So I think pfn or kva aren't set correctly.
>>>>> >> We tried to boot domU with revesing cset xen-ia64-ustable.8790
>>>>> >> and it was good work.
>>>>> >>
>>>>> >> I'm still debugging it. :-<
>>>>> >>
>>>>> >> Best Regards,
>>>>> >>
>>>>> >> Akio Takebe
>>>>> >>
>>>>> >> >Confirmed cset xen-unstable 8783 fails while 8782 succeeds.
>>>>> >> >
>>>>> >> >Perhaps there's something different about mmap on RH
>>>>> >> >vs Suse and Debian? Perhaps only on ia64?
>>>>> >> >
>>>>> >>
>>>>> >>
>>>>> >>
>>>>>
>>>>>
>>>>>
>>>>
>>>>_______________________________________________
>>>>Xen-devel mailing list
>>>>Xen-devel@lists.xensource.com
>>>>http://lists.xensource.com/xen-devel
>>
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>
>_______________________________________________
>Xen-ia64-devel mailing list
>Xen-ia64-devel@lists.xensource.com
>http://lists.xensource.com/xen-ia64-devel
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Xen-devel] Help? Red Hat fails, Suse/Debian both work fine
2006-03-02 5:43 Help? Red Hat fails, Suse/Debian both work fine Tian, Kevin
2006-03-02 6:04 ` [Xen-ia64-devel] " Yoshi.Oguchi
@ 2006-03-02 8:26 ` Keir Fraser
2006-03-02 15:18 ` Akio Takebe
2 siblings, 0 replies; 8+ messages in thread
From: Keir Fraser @ 2006-03-02 8:26 UTC (permalink / raw)
To: Tian, Kevin; +Cc: xen-devel, xen-ia64-devel
On 2 Mar 2006, at 05:43, Tian, Kevin wrote:
> However the interesting thing is, following Cset is only for
> changing way to map dom0's store page, instead of domU. DomU's
> store page is still mapped by foreign page map. If above hint is real
> cause, xend start can fail earlier due to incorrect mapping when
> introducing dom0 into xenstore. However you all observe the bug
> bothering only when domU is booting...
The dom0 connection is only used in anger when first creating a domU,
to set up device backend state. Until that time it could be screwed and
you probably would not notice.
-- Keir
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: Help? Red Hat fails, Suse/Debian both work fine
2006-03-02 5:43 Help? Red Hat fails, Suse/Debian both work fine Tian, Kevin
2006-03-02 6:04 ` [Xen-ia64-devel] " Yoshi.Oguchi
2006-03-02 8:26 ` [Xen-devel] " Keir Fraser
@ 2006-03-02 15:18 ` Akio Takebe
2006-03-02 17:24 ` [Xen-ia64-devel] " Alex Williamson
2 siblings, 1 reply; 8+ messages in thread
From: Akio Takebe @ 2006-03-02 15:18 UTC (permalink / raw)
To: Tian, Kevin, Xu, Anthony, Magenheimer, Dan (HP Labs Fort Collins),
xen-devel
Cc: yo.fujita, Akio Takebe, xen-ia64-devel
Hi, Kevin
Thank you for your advice.
I tried to compile define ARCH_HAS_DEV_MEM.
but nothing changed.
Best Regards,
Akio Takebe
>Hi, Akio,
> Currently linux-2.6-xen-sparse/driver/xen/char/ is not included
>in compilation for xen/ia64, so you're still using linux-style kmem
>path.
>
> Try to compile that directory into your xenlinux image, and
>define ARCH_HAS_DEV_MEM, and then see anything changed
>for you.
>
> However the interesting thing is, following Cset is only for
>changing way to map dom0's store page, instead of domU. DomU's
>store page is still mapped by foreign page map. If above hint is real
>cause, xend start can fail earlier due to incorrect mapping when
>introducing dom0 into xenstore. However you all observe the bug
>bothering only when domU is booting...
>
>Thanks,
>Kevin
>
>>-----Original Message-----
>>From: xen-devel-bounces@lists.xensource.com
>>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Akio Takebe
>>Sent: 2006年3月2日 8:44
>>To: Xu, Anthony; Magenheimer, Dan (HP Labs Fort Collins); xen-devel
>>Cc: yo.fujita; Akio Takebe; xen-ia64-devel@lists.xensource.com
>>Subject: RE: [Xen-devel] Help? Red Hat fails, Suse/Debian both work fine
>>
>>Hi, Anthony
>>
>>Thank you for your advice.
>>I checked retun value of mmap(), and it is not NULL.
>>I'll check vcpu_translate().
>>
>>Best Regards,
>>
>>Akio Takebe
>>
>>>>It is likely some subtle difference (or bug), perhaps in mmap?
>>>
>>>As I know, in Redhat, mmap can return NULL address, but seems xen
>>>can't handle this situation, see below code segment:
>>>
>>>In function vcpu_translate() of vcpu.c file
>>>
>>> else if (!region && warn_region0_address) {
>>> REGS *regs = vcpu_regs(vcpu);
>>> unsigned long viip = PSCB(vcpu,iip);
>>> unsigned long vipsr = PSCB(vcpu,ipsr);
>>> unsigned long iip = regs->cr_iip;
>>> unsigned long ipsr = regs->cr_ipsr;
>>> printk("vcpu_translate: bad address %p, viip=%p, vipsr=%p, iip=%p,
>>>ipsr=%p continuing\n", address, viip, vipsr, iip, ipsr);
>>> }
>>>warn_region0_address is turned off by default,
>>>so maybe we can turn on warn_region0_address to see whether this is the
>>>root cause.
>>>
>>>Thanks,
>>>-Anthony
>>>
>>>>-----Original Message-----
>>>>From: xen-devel-bounces@lists.xensource.com
>>>>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of
>>Magenheimer, Dan
>>>>(HP Labs Fort Collins)
>>>>Sent: 2006ト・ヤツ1ネユ 5:45
>>>>To: xen-devel
>>>>Cc: yo.fujita; Akio Takebe; xen-ia64-devel@lists.xensource.com
>>>>Subject: [Xen-devel] Help? Red Hat fails, Suse/Debian both work fine
>>>>
>>>>Hi all --
>>>>
>>>>We are seeing a strange problem where a recent cset causes
>>>>Red Hat to fail domU boot on ia64 complaining of a hotplug
>>>>problem but doesn't cause any problem for Suse or Debian.
>>>>It is likely some subtle difference (or bug), perhaps in mmap?
>>>>Suggestions/advice from anyone more familiar with distro
>>>>differences (on ia64) would be appreciated.
>>>>
>>>>Changeset is xen-unstable 8783 ("Use /dev/kmem to map dom0
>>>>xenstore page instead of abusing the foreign mapping interface.",
>>>>Feb 8, committed by Christian). Backing out this changeset
>>>>or using the small patch below causes the problem to go away,
>>>>so we have a workaround, but a root cause would be nice
>>>>to know and fix.
>>>>
>>>>Full thread of discussion can be found here:
>>>>http://lists.xensource.com/archives/html/xen-ia64-devel/2006-02/msg00104
>>>>.html
>>>>
>>>>Thanks,
>>>>Dan
>>>>
>>>>> -----Original Message-----
>>>>> From: Akio Takebe [mailto:takebe_akio@jp.fujitsu.com]
>>>>> Sent: Tuesday, February 28, 2006 2:47 AM
>>>>> To: Magenheimer, Dan (HP Labs Fort Collins); yo.fujita;
>>>>> xen-ia64-devel@lists.xensource.com
>>>>> Cc: Akio Takebe
>>>>> Subject: RE: [Xen-ia64-devel] Weekly benchmark results [2/3rd week]
>>>>>
>>>>> Hi, Dan
>>>>>
>>>>> I'm still debuging, but it is very difficult...
>>>>> Much advice is welcome. :-)
>>>>>
>>>>> Now I can boot domU by using the following patch.
>>>>>
>>>>> diff -r 6c43118bdba8 tools/xenstore/xenstored_domain.c
>>>>> --- a/tools/xenstore/xenstored_domain.c Fri Feb 24 15:41:08 2006 -0700
>>>>> +++ b/tools/xenstore/xenstored_domain.c Tue Feb 28 18:20:16 2006
>>+0900
>>>>> @@ -467,6 +467,7 @@ static int dom0_init(void)
>>>>> int rc, fd;
>>>>> evtchn_port_t port;
>>>>> unsigned long kva;
>>>>> + unsigned long mfn;
>>>>> char str[20];
>>>>> struct domain *dom0;
>>>>>
>>>>> @@ -500,9 +501,16 @@ static int dom0_init(void)
>>>>> if (fd == -1)
>>>>> return -1;
>>>>>
>>>>> - dom0->interface = mmap(NULL, getpagesize(),
>>>>> PROT_READ|PROT_WRITE,
>>>>> - MAP_SHARED, fd, kva);
>>>>> - if (dom0->interface == MAP_FAILED)
>>>>> + mfn=((0x0fffffffffffffff & kva) >>14);
>>>>> +/*
>>>>> + dom0->interface = mmap(NULL, getpagesize(),
>>>>> PROT_READ|PROT_WRITE,
>>>>> + MAP_SHARED, fd, kva);
>>>>> +*/
>>>>> + dom0->interface = xc_map_foreign_range(
>>>>> + *xc_handle, 0,
>>>>> + getpagesize(), PROT_READ|PROT_WRITE, mfn);
>>>>> + if (!dom0->interface)
>>>>> +// if (dom0->interface == MAP_FAILED)
>>>>> goto outfd;
>>>>>
>>>>> close(fd);
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Akio Takebe
>>>>>
>>>>> >Hi Akio --
>>>>> >
>>>>> >Any more progress on this issue? If you are stuck,
>>>>> >maybe we should post the problem to xen-devel to
>>>>> >see if we can get help from a Red Hat person (since
>>>>> >the problem doesn't occur on Suse or Debian).
>>>>> >
>>>>> >Thanks,
>>>>> >Dan
>>>>> >
>>>>> >> -----Original Message-----
>>>>> >> From: Akio Takebe [mailto:takebe_akio@jp.fujitsu.com]
>>>>> >> Sent: Thursday, February 23, 2006 8:45 PM
>>>>> >> To: Magenheimer, Dan (HP Labs Fort Collins); yo.fujita;
>>>>> >> xen-ia64-devel@lists.xensource.com
>>>>> >> Cc: Akio Takebe
>>>>> >> Subject: RE: [Xen-ia64-devel] Weekly benchmark results [2/3rd week]
>>>>> >>
>>>>> >> Hi, Dan and Alex
>>>>> >>
>>>>> >> I think this issue is only on ia64.
>>>>> >> I seem that kmem_map@drivers/char/mem.c is used on ia64,
>>>>> >> but mem_map@drivers/xen/char/mem.c is used on x86.
>>>>> >> So I think pfn or kva aren't set correctly.
>>>>> >> We tried to boot domU with revesing cset xen-ia64-ustable.8790
>>>>> >> and it was good work.
>>>>> >>
>>>>> >> I'm still debugging it. :-<
>>>>> >>
>>>>> >> Best Regards,
>>>>> >>
>>>>> >> Akio Takebe
>>>>> >>
>>>>> >> >Confirmed cset xen-unstable 8783 fails while 8782 succeeds.
>>>>> >> >
>>>>> >> >Perhaps there's something different about mmap on RH
>>>>> >> >vs Suse and Debian? Perhaps only on ia64?
>>>>> >> >
>>>>> >>
>>>>> >>
>>>>> >>
>>>>>
>>>>>
>>>>>
>>>>
>>>>_______________________________________________
>>>>Xen-devel mailing list
>>>>Xen-devel@lists.xensource.com
>>>>http://lists.xensource.com/xen-devel
>>
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Xen-ia64-devel] RE: Help? Red Hat fails, Suse/Debian both work fine
2006-03-02 15:18 ` Akio Takebe
@ 2006-03-02 17:24 ` Alex Williamson
2006-03-02 18:41 ` Ewan Mellor
0 siblings, 1 reply; 8+ messages in thread
From: Alex Williamson @ 2006-03-02 17:24 UTC (permalink / raw)
To: Akio Takebe
Cc: Magenheimer, Dan (HP Labs Fort Collins), Tian, Kevin, Xu, Anthony,
xen-devel, xen-ia64-devel
A little bit of data in case it sparks some debug ideas. Enabling
tracing in xenstored shows that the difference between the working and
non-working case are exhibited pretty quickly. There's definitely
communication occurring through the mmap'd kmem page, but it's missing
some important chunks. For instance, here's the first few messages
logged in the working case:
CREATE connection 0x6000000000022678
IN 0x6000000000022678 07:15:03 DIRECTORY (device )
OUT 0x6000000000022678 07:15:03 ERROR (ENOENT )
IN 0x6000000000022678 07:15:03 DIRECTORY (backend )
OUT 0x6000000000022678 07:15:03 ERROR (ENOENT )
IN 0x6000000000022678 07:15:03 WATCH (device A000000100D20520 )
CREATE watch 0x60000000000229a8
OUT 0x6000000000022678 07:15:03 WATCH (OK )
IN 0x6000000000022678 07:15:03 WATCH (backend A000000100D20500 )
CREATE watch 0x60000000000228c8
OUT 0x6000000000022678 07:15:03 WATCH_EVENT (device A000000100D20520 )
OUT 0x6000000000022678 07:15:03 WATCH (OK )
OUT 0x6000000000022678 07:15:03 WATCH_EVENT (backend A000000100D20500 )
CREATE connection 0x6000000000022748
IN 0x6000000000022748 07:15:03 WATCH (@introduceDomain domlist )
CREATE watch 0x60000000000234c8
OUT 0x6000000000022748 07:15:03 WATCH (OK )
In the failing case, I see:
CREATE connection 0x6000000000026388
IN 0x6000000000026388 07:07:47 DIRECTORY (device )
OUT 0x6000000000026388 07:07:47 ERROR (ENOENT )
CREATE connection 0x60000000000265d8
IN 0x60000000000265d8 07:07:47 WATCH (@introduceDomain domlist )
CREATE watch 0x6000000000026d78
OUT 0x60000000000265d8 07:07:47 WATCH (OK )
Note that only the lines dealing with A000000100D20520 are missing.
The next absent chunk comes after about 300 lines of identical trace
between the working and failing case. This block is missing when it
fails:
OUT 0x6000000000022678 07:15:14 WATCH_EVENT (backend/vbd/1/769 A000000100D20500 )
IN 0x6000000000022678 07:15:14 DIRECTORY (backend/vbd/1/769 )
OUT 0x6000000000022678 07:15:14 WATCH_EVENT (backend/vbd/1/769/domain A000000100D20500 )
OUT 0x6000000000022678 07:15:14 WATCH_EVENT (backend/vbd/1/769/frontend A000000100D20500 )
OUT 0x6000000000022678 07:15:14 WATCH_EVENT (backend/vbd/1/769/dev A000000100D20500 )
OUT 0x6000000000022678 07:15:14 WATCH_EVENT (backend/vbd/1/769/state A000000100D20500 )
OUT 0x6000000000022678 07:15:14 WATCH_EVENT (backend/vbd/1/769/params A000000100D20500 )
OUT 0x6000000000022678 07:15:14 WATCH_EVENT (backend/vbd/1/769/mode A000000100D20500 )
OUT 0x6000000000022678 07:15:14 WATCH_EVENT (backend/vbd/1/769/frontend-id A000000100D20500 )
OUT 0x6000000000022678 07:15:14 WATCH_EVENT (backend/vbd/1/769/type A000000100D20500 )
OUT 0x6000000000022678 07:15:14 DIRECTORY (domain frontend dev state params mode frontend-id type )
IN 0x6000000000022678 07:15:14 READ (backend/vbd/1/769/state )
OUT 0x6000000000022678 07:15:14 READ (1)
IN 0x6000000000022678 07:15:14 READ (backend/vbd/1/769/frontend-id )
OUT 0x6000000000022678 07:15:14 READ (1)
IN 0x6000000000022678 07:15:14 READ (backend/vbd/1/769/frontend )
OUT 0x6000000000022678 07:15:14 READ (/local/domain/1/device/vbd/769)
IN 0x6000000000022678 07:15:14 DIRECTORY (/local/domain/1/device/vbd/769 )
OUT 0x6000000000022678 07:15:14 DIRECTORY (virtual-device backend-id state backend )
IN 0x6000000000022678 07:15:14 READ (backend/vbd/1/769/frontend-id )
OUT 0x6000000000022678 07:15:14 READ (1)
IN 0x6000000000022678 07:15:14 READ (backend/vbd/1/769/frontend )
OUT 0x6000000000022678 07:15:14 READ (/local/domain/1/device/vbd/769)
IN 0x6000000000022678 07:15:14 DIRECTORY (/local/domain/1/device/vbd/769 )
OUT 0x6000000000022678 07:15:14 DIRECTORY (virtual-device backend-id state backend )
IN 0x6000000000022678 07:15:14 WATCH (backend/vbd/1/769/physical-device E000000008E31C90 )
CREATE watch 0x600000000000cad8
OUT 0x6000000000022678 07:15:14 WATCH (OK )
IN 0x6000000000022678 07:15:14 READ (backend/vbd/1/769/state )
OUT 0x6000000000022678 07:15:14 WATCH_EVENT (backend/vbd/1/769/physical-device E000000008E31C90 )
OUT 0x6000000000022678 07:15:14 READ (1)
IN 0x6000000000022678 07:15:14 WRITE (backend/vbd/1/769/state 2)
OUT 0x6000000000022678 07:15:14 WATCH_EVENT (backend/vbd/1/769/state A000000100D20500 )
OUT 0x6000000000022678 07:15:14 WRITE (OK )
IN 0x6000000000022678 07:15:14 WATCH (/local/domain/1/device/vbd/769/state E00000001D3AB1A0 )
CREATE watch 0x600000000000cda8
OUT 0x6000000000022678 07:15:14 WATCH (OK )
IN 0x6000000000022678 07:15:14 DIRECTORY (backend/vbd/1/769/domain )
OUT 0x6000000000022678 07:15:14 WATCH_EVENT (/local/domain/1/device/vbd/769/state E00000001D3AB1A0 )
OUT 0x6000000000022678 07:15:14 DIRECTORY ()
IN 0x6000000000022678 07:15:14 DIRECTORY (backend/vbd/1/769/frontend )
OUT 0x6000000000022678 07:15:14 DIRECTORY ()
IN 0x6000000000022678 07:15:14 DIRECTORY (backend/vbd/1/769/dev )
OUT 0x6000000000022678 07:15:14 DIRECTORY ()
IN 0x6000000000022678 07:15:14 DIRECTORY (backend/vbd/1/769/state )
OUT 0x6000000000022678 07:15:14 DIRECTORY ()
IN 0x6000000000022678 07:15:14 DIRECTORY (backend/vbd/1/769/params )
OUT 0x6000000000022678 07:15:14 DIRECTORY ()
IN 0x6000000000022678 07:15:14 DIRECTORY (backend/vbd/1/769/mode )
OUT 0x6000000000022678 07:15:14 DIRECTORY ()
IN 0x6000000000022678 07:15:14 DIRECTORY (backend/vbd/1/769/frontend-id )
OUT 0x6000000000022678 07:15:14 DIRECTORY ()
IN 0x6000000000022678 07:15:14 DIRECTORY (backend/vbd/1/769/type )
OUT 0x6000000000022678 07:15:14 DIRECTORY ()
IN 0x6000000000022678 07:15:14 READ (backend/vbd/1/769/physical-device )
OUT 0x6000000000022678 07:15:14 ERROR (ENOENT )
IN 0x6000000000022678 07:15:14 DIRECTORY (backend/vbd/1/769/state )
OUT 0x6000000000022678 07:15:14 DIRECTORY ()
IN 0x6000000000022678 07:15:14 READ (/local/domain/1/device/vbd/769/state )
OUT 0x6000000000022678 07:15:14 READ (1)
If I mmap and dump the kva page in a test program, it looks like it's
working correctly. I can see and decode the send and receive rings and
see the index values. Thanks,
Alex
--
Alex Williamson HP Linux & Open Source Lab
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Xen-ia64-devel] RE: Help? Red Hat fails, Suse/Debian both work fine
2006-03-02 17:24 ` [Xen-ia64-devel] " Alex Williamson
@ 2006-03-02 18:41 ` Ewan Mellor
2006-03-02 18:56 ` Alex Williamson
0 siblings, 1 reply; 8+ messages in thread
From: Ewan Mellor @ 2006-03-02 18:41 UTC (permalink / raw)
To: Alex Williamson
Cc: Tian, Kevin, xen-devel, Akio Takebe,
Magenheimer, Dan (HP Labs Fort Collins), Xu, Anthony,
xen-ia64-devel
On Thu, Mar 02, 2006 at 10:24:00AM -0700, Alex Williamson wrote:
>
> A little bit of data in case it sparks some debug ideas. Enabling
> tracing in xenstored shows that the difference between the working and
> non-working case are exhibited pretty quickly. There's definitely
> communication occurring through the mmap'd kmem page, but it's missing
> some important chunks. For instance, here's the first few messages
> logged in the working case:
>
> CREATE connection 0x6000000000022678
> IN 0x6000000000022678 07:15:03 DIRECTORY (device )
> OUT 0x6000000000022678 07:15:03 ERROR (ENOENT )
> IN 0x6000000000022678 07:15:03 DIRECTORY (backend )
> OUT 0x6000000000022678 07:15:03 ERROR (ENOENT )
> IN 0x6000000000022678 07:15:03 WATCH (device A000000100D20520 )
> CREATE watch 0x60000000000229a8
> OUT 0x6000000000022678 07:15:03 WATCH (OK )
> IN 0x6000000000022678 07:15:03 WATCH (backend A000000100D20500 )
> CREATE watch 0x60000000000228c8
> OUT 0x6000000000022678 07:15:03 WATCH_EVENT (device A000000100D20520 )
> OUT 0x6000000000022678 07:15:03 WATCH (OK )
> OUT 0x6000000000022678 07:15:03 WATCH_EVENT (backend A000000100D20500 )
> CREATE connection 0x6000000000022748
> IN 0x6000000000022748 07:15:03 WATCH (@introduceDomain domlist )
> CREATE watch 0x60000000000234c8
> OUT 0x6000000000022748 07:15:03 WATCH (OK )
>
> In the failing case, I see:
>
> CREATE connection 0x6000000000026388
> IN 0x6000000000026388 07:07:47 DIRECTORY (device )
> OUT 0x6000000000026388 07:07:47 ERROR (ENOENT )
> CREATE connection 0x60000000000265d8
> IN 0x60000000000265d8 07:07:47 WATCH (@introduceDomain domlist )
> CREATE watch 0x6000000000026d78
> OUT 0x60000000000265d8 07:07:47 WATCH (OK )
>
> Note that only the lines dealing with A000000100D20520 are missing.
These are requests from the xenbus driver to create watches to monitor for new
devices being created. That presumably means that either your xenbus driver
is fubar'd, or the mmap'd page is bust, as discussed earlier.
The requests that you are seeing are from xenconsoled and xend, each of which
is using the unix domain socket to talk to the store, not the shared page.
> The next absent chunk comes after about 300 lines of identical trace
> between the working and failing case. This block is missing when it
> fails:
None of the watches are firing, because they haven't been registered. No
surprise there.
Ewan.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Xen-ia64-devel] RE: Help? Red Hat fails, Suse/Debian both work fine
2006-03-02 18:41 ` Ewan Mellor
@ 2006-03-02 18:56 ` Alex Williamson
2006-03-02 20:27 ` RE: [Xen-devel] " Keir Fraser
0 siblings, 1 reply; 8+ messages in thread
From: Alex Williamson @ 2006-03-02 18:56 UTC (permalink / raw)
To: Ewan Mellor
Cc: Tian, Kevin, xen-devel, Akio Takebe,
Magenheimer, Dan (HP Labs Fort Collins), Xu, Anthony,
xen-ia64-devel
On Thu, 2006-03-02 at 18:41 +0000, Ewan Mellor wrote:
>
> These are requests from the xenbus driver to create watches to monitor for new
> devices being created. That presumably means that either your xenbus driver
> is fubar'd, or the mmap'd page is bust, as discussed earlier.
>
> The requests that you are seeing are from xenconsoled and xend, each of which
> is using the unix domain socket to talk to the store, not the shared page.
If I dump the xsd_kva page using mmap from another app, there are a
few entries made:
0000: 01 00 00 00 00 00 00 00 00 00 00 00 07 00 00 00
0010: 64 65 76 69 63 65 00 00 00 00 00 00 00 00 00 00
0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
...
0400: 10 00 00 00 00 00 00 00 00 00 00 00 07 00 00 00
0410: 45 4e 4f 45 4e 54 00 00 00 00 00 00 00 00 00 00
0420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
...
0800: 17 00 00 00 17 00 00 00 00 00 00 00 17 00 00 00
As you indicate, lots of output in the xenstored-trace file, but hardly
anything here. We are getting some transactions in there though, so
maybe we're dealing with a memory ordering issue. Thanks,
Alex
--
Alex Williamson HP Linux & Open Source Lab
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: RE: [Xen-devel] Help? Red Hat fails, Suse/Debian both work fine
2006-03-02 18:56 ` Alex Williamson
@ 2006-03-02 20:27 ` Keir Fraser
0 siblings, 0 replies; 8+ messages in thread
From: Keir Fraser @ 2006-03-02 20:27 UTC (permalink / raw)
To: Alex Williamson; +Cc: xen-devel, Ewan Mellor, xen-ia64-devel
On 2 Mar 2006, at 18:56, Alex Williamson wrote:
> If I dump the xsd_kva page using mmap from another app, there are a
> few entries made:
>
> 0000: 01 00 00 00 00 00 00 00 00 00 00 00 07 00 00 00
> 0010: 64 65 76 69 63 65 00 00 00 00 00 00 00 00 00 00
> 0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> ...
> 0400: 10 00 00 00 00 00 00 00 00 00 00 00 07 00 00 00
> 0410: 45 4e 4f 45 4e 54 00 00 00 00 00 00 00 00 00 00
> 0420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> ...
> 0800: 17 00 00 00 17 00 00 00 00 00 00 00 17 00 00 00
>
> As you indicate, lots of output in the xenstored-trace file, but hardly
> anything here. We are getting some transactions in there though, so
> maybe we're dealing with a memory ordering issue. Thanks,
There could be stuff in the page because the kernel has written
requests there, but it doesn't necessarily mean they have been picked
up by xenstored. Occam's razor says this page is not mapped properly
into xenstored's address space: I think it's desperately unlikely to be
something as subtle as memory ordering. As Ewan noted, any xenstore
activity is probably driven by clients connected via a unix domain
socket, not via domain0's xenbus page.
-- Keir
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2006-03-02 20:27 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2006-03-02 5:43 Help? Red Hat fails, Suse/Debian both work fine Tian, Kevin
2006-03-02 6:04 ` [Xen-ia64-devel] " Yoshi.Oguchi
2006-03-02 8:26 ` [Xen-devel] " Keir Fraser
2006-03-02 15:18 ` Akio Takebe
2006-03-02 17:24 ` [Xen-ia64-devel] " Alex Williamson
2006-03-02 18:41 ` Ewan Mellor
2006-03-02 18:56 ` Alex Williamson
2006-03-02 20:27 ` RE: [Xen-devel] " Keir Fraser
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.