mapped memory delay?

mapped memory delay?

[Xin Zhao]

I ran into the following problem:

I setup a shared ring page and an event channel between two domains (dom0
and one domU), DomU increases req_prod by one and notify the dom0 via the
event channel, dom0 then check req_prod pointer and process the request in
the shared ring. But I noticed that sometimes dom0 get the irq triggered
by the notification event, but the req_prod remains the same. This
situation only happened occasionally. Why? Because the req_prod is saved
in the shared, mapped page, I suspect that there is a delay of updates
between two domains. Is that true? If so, how to handle this problem?

Your advice is highly appreciated!

-x


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: mapped memory delay?

[Christopher Clark]

Are you running this on a single processor core?

If not, it sounds to me like a cache coherency issue - you'll need a
write memory barrier after writing to the req_prod.  If you're using
the ring macros from io/ring.h this should be already taken care of
for you in the push macros.

c.


On Fri, 18 Feb 2005 16:44:30 -0500 (EST), Xin Zhao
<zhaoxin@eecs.umich.edu> wrote:
> I ran into the following problem:
> 
> I setup a shared ring page and an event channel between two domains (dom0
> and one domU), DomU increases req_prod by one and notify the dom0 via the
> event channel, dom0 then check req_prod pointer and process the request in
> the shared ring. But I noticed that sometimes dom0 get the irq triggered
> by the notification event, but the req_prod remains the same. This
> situation only happened occasionally. Why? Because the req_prod is saved
> in the shared, mapped page, I suspect that there is a delay of updates
> between two domains. Is that true? If so, how to handle this problem?
> 
> Your advice is highly appreciated!
> 
> -x
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
>


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: mapped memory delay?

[Xin Zhao]

Thanks for your advice. I tried rmb() and wmb() before I sent last email,
they does not help.

Actually I am running a single processor core. :( So it should not have
any delay. But I still ran into the problem. Another interesting thing is
that my working flow should be as follows:

domU issue a request -> generate an irq at dom0 -> domU waits for reply->
dom0 gets the request, processes it and reply -> domU wakes up by irq
caused by notification event sent by dom0, gets the result -> DomU sends
another request

That means, if domU does not get the irq from dom0, it will not issue next
request. Because of the problem of req_prod updates, dom0 failed to get
the request and thus will not process it, not mention to reply. So dom0
should not be able to receive next request or next irq.

But the fact is that dom0 can still get irq even it failed to get last
request. It appears that dom0 issues multiple irqs for one event. How
could that happed?

-x


On Fri, 18 Feb 2005, Christopher Clark wrote:

> Are you running this on a single processor core?
>
> If not, it sounds to me like a cache coherency issue - you'll need a
> write memory barrier after writing to the req_prod.  If you're using
> the ring macros from io/ring.h this should be already taken care of
> for you in the push macros.
>
> c.
>
>
> On Fri, 18 Feb 2005 16:44:30 -0500 (EST), Xin Zhao
> <zhaoxin@eecs.umich.edu> wrote:
> > I ran into the following problem:
> >
> > I setup a shared ring page and an event channel between two domains (dom0
> > and one domU), DomU increases req_prod by one and notify the dom0 via the
> > event channel, dom0 then check req_prod pointer and process the request in
> > the shared ring. But I noticed that sometimes dom0 get the irq triggered
> > by the notification event, but the req_prod remains the same. This
> > situation only happened occasionally. Why? Because the req_prod is saved
> > in the shared, mapped page, I suspect that there is a delay of updates
> > between two domains. Is that true? If so, how to handle this problem?
> >
> > Your advice is highly appreciated!
> >
> > -x
> >
> > -------------------------------------------------------
> > SF email is sponsored by - The IT Product Guide
> > Read honest & candid reviews on hundreds of IT Products from real users.
> > Discover which products truly live up to the hype. Start reading now.
> > http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> >
>


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: mapped memory delay?

[Christopher Clark]

This sounds like the symptoms you'd experience with multiple functions
using the same event channel port.

How are you obtaining the ports you're using to communicate?
Are you using alloc_unbound ?

c.

On Fri, 2005-02-18 at 22:41, Xin Zhao wrote:
> Thanks for your advice. I tried rmb() and wmb() before I sent last email,
> they does not help.
> 
> Actually I am running a single processor core. :( So it should not have
> any delay. But I still ran into the problem. Another interesting thing is
> that my working flow should be as follows:
> 
> domU issue a request -> generate an irq at dom0 -> domU waits for reply->
> dom0 gets the request, processes it and reply -> domU wakes up by irq
> caused by notification event sent by dom0, gets the result -> DomU sends
> another request
> 
> That means, if domU does not get the irq from dom0, it will not issue next
> request. Because of the problem of req_prod updates, dom0 failed to get
> the request and thus will not process it, not mention to reply. So dom0
> should not be able to receive next request or next irq.
> 
> But the fact is that dom0 can still get irq even it failed to get last
> request. It appears that dom0 issues multiple irqs for one event. How
> could that happed?
> 
> -x
> 
> 
> On Fri, 18 Feb 2005, Christopher Clark wrote:
> 
> > Are you running this on a single processor core?
> >
> > If not, it sounds to me like a cache coherency issue - you'll need a
> > write memory barrier after writing to the req_prod.  If you're using
> > the ring macros from io/ring.h this should be already taken care of
> > for you in the push macros.
> >
> > c.
> >
> >
> > On Fri, 18 Feb 2005 16:44:30 -0500 (EST), Xin Zhao
> > <zhaoxin@eecs.umich.edu> wrote:
> > > I ran into the following problem:
> > >
> > > I setup a shared ring page and an event channel between two domains (dom0
> > > and one domU), DomU increases req_prod by one and notify the dom0 via the
> > > event channel, dom0 then check req_prod pointer and process the request in
> > > the shared ring. But I noticed that sometimes dom0 get the irq triggered
> > > by the notification event, but the req_prod remains the same. This
> > > situation only happened occasionally. Why? Because the req_prod is saved
> > > in the shared, mapped page, I suspect that there is a delay of updates
> > > between two domains. Is that true? If so, how to handle this problem?
> > >
> > > Your advice is highly appreciated!
> > >
> > > -x
> > >
> > > -------------------------------------------------------
> > > SF email is sponsored by - The IT Product Guide
> > > Read honest & candid reviews on hundreds of IT Products from real users.
> > > Discover which products truly live up to the hype. Start reading now.
> > > http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/xen-devel
> > >
> >



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: mapped memory delay?

[Mark Williamson]

> I would like to do the same thing, right now I have shared mem between
> domU and dom0, but no good syncronization. Is there an example somewhere
> of how to do this in the new xcs world? It should be easier now that
> this code has been separated out, right?

If you're sharing memory directly between the two then you don't need to 
involve xcs (other than in the setup phase) since that just does control 
messages.

You might find the ring macros in xen/include/public/ring.h useful.  They make 
it easy to declare ring datatypes (with appropriately sized rings) and 
maintain producer / consumer indexes, with the appropriate memory barriers.

Examples of use are in linux-2.6.10-xen-sparse/drivers/xen/{blk,usb}
{back,front}.  (the net drivers use a slightly different ring format 
instead).

Cheers,
Mark


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: mapped memory delay?

[Jacob Gorm Hansen]

Xin Zhao wrote:
> I ran into the following problem:
> 
> I setup a shared ring page and an event channel between two domains (dom0
> and one domU), DomU increases req_prod by one and notify the dom0 via the
> event channel, dom0 then check req_prod pointer and process the request in
> the shared ring. But I noticed that sometimes dom0 get the irq triggered
> by the notification event, but the req_prod remains the same. This
> situation only happened occasionally.

hi,

I would like to do the same thing, right now I have shared mem between 
domU and dom0, but no good syncronization. Is there an example somewhere 
of how to do this in the new xcs world? It should be easier now that 
this code has been separated out, right?

thanks,
Jacob


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: mapped memory delay?

[Mark Williamson]

> Hmm will this be any different if the guy I wish to share this with in
> dom0 is a userspace process? Can I use /dev/evtchn for that, or do I
> have to cook my own device node for this?

I think nowadays there can be multiple listeners on /dev/evtchn (as long as, 
obviously, they're all listening to different things), so yes that should 
work fine.  Of course, I'm not sure anyone's tried this...

Cheers,
Mark


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: mapped memory delay?

[Jacob Gorm Hansen]

Mark Williamson wrote:
> If you're sharing memory directly between the two then you don't need to 
> involve xcs (other than in the setup phase) since that just does control 
> messages.
> 
> You might find the ring macros in xen/include/public/ring.h useful.  They make 
> it easy to declare ring datatypes (with appropriately sized rings) and 
> maintain producer / consumer indexes, with the appropriate memory barriers.
> 
> Examples of use are in linux-2.6.10-xen-sparse/drivers/xen/{blk,usb}
> {back,front}.  (the net drivers use a slightly different ring format 
> instead).

Hmm will this be any different if the guy I wish to share this with in 
dom0 is a userspace process? Can I use /dev/evtchn for that, or do I 
have to cook my own device node for this?

thanks,

Jacob


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: mapped memory delay?

[Anthony Liguori]

Mark Williamson wrote:

>I think nowadays there can be multiple listeners on /dev/evtchn (as long as, 
>obviously, they're all listening to different things), so yes that should 
>work fine.  Of course, I'm not sure anyone's tried this...
>  
>
I have.  It worked fine for me.

>Cheers,
>Mark
>
>
>-------------------------------------------------------
>SF email is sponsored by - The IT Product Guide
>Read honest & candid reviews on hundreds of IT Products from real users.
>Discover which products truly live up to the hype. Start reading now.
>http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/xen-devel
>
>  
>


-- 
Anthony Liguori
anthony@codemonkey.ws



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: mapped memory delay?

[Jacob Gorm Hansen]

Anthony Liguori wrote:
> Mark Williamson wrote:
> 
>> I think nowadays there can be multiple listeners on /dev/evtchn (as 
>> long as, obviously, they're all listening to different things), so yes 
>> that should work fine.  Of course, I'm not sure anyone's tried this...
>>  
>>
> I have.  It worked fine for me.

Just to be lazy: do you have any code you could post? Do I just open the 
device and read whatever is in there, or how does it work?

Jacob


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: mapped memory delay?

[Jacob Gorm Hansen]

Jacob Gorm Hansen wrote:

> Just to be lazy: do you have any code you could post? Do I just open the 
> device and read whatever is in there, or how does it work?

Never mind, doesn't look to hard ;-)

Jacob


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click