use xs_watch from userspace and use epoll instead of select

use xs_watch from userspace and use epoll instead of select

[Vasiliy G Tolstov]

Hello. I have some examples of usage xs_watch in userspace that contains
loop with select function usage. If i use epoll, no events returned from
epoll_wait...
Does it matter to use select from fd that used for watch?
epoll fd added with ev.events = (EPOLLIN | EPOLLPRI | EPOLLONESHOT);


-- 
Vasiliy G Tolstov <v.tolstov@selfip.ru>
Selfip.Ru

Re: use xs_watch from userspace and use epoll instead of select

[Ian Campbell]

On Fri, 2011-06-03 at 13:04 +0100, Vasiliy G Tolstov wrote:
> Hello. I have some examples of usage xs_watch in userspace that contains
> loop with select function usage. If i use epoll, no events returned from
> epoll_wait...
> Does it matter to use select from fd that used for watch?
> epoll fd added with ev.events = (EPOLLIN | EPOLLPRI | EPOLLONESHOT);

I think you need to provide an example of the code which doesn't work.

Ian.

Re: use xs_watch from userspace and use epoll instead of select

[Vasiliy G Tolstov]

On Fri, 2011-06-03 at 13:13 +0100, Ian Campbell wrote:
> On Fri, 2011-06-03 at 13:04 +0100, Vasiliy G Tolstov wrote:
> > Hello. I have some examples of usage xs_watch in userspace that contains
> > loop with select function usage. If i use epoll, no events returned from
> > epoll_wait...
> > Does it matter to use select from fd that used for watch?
> > epoll fd added with ev.events = (EPOLLIN | EPOLLPRI | EPOLLONESHOT);
> 
> I think you need to provide an example of the code which doesn't work.
> 
> Ian.
> 

This is not work:
  if ((err = xs_watch(stats->xs, "data/xenmgm", "xenmgm")) != 0) {
    log_error("failed to create xenstore watch on path: %s, error: %s",
"data/xenmgm", strerror(err));
  }
  fd = xs_fileno(stats->xs);
  xen_events = malloc(sizeof(struct epoll_event));
  memset(xen_events, 0, sizeof(struct epoll_event));
  if ((epfd = epoll_create(1)) < 0) {
    log_error("%s", "failed epoll_create");
    return -1;
  }
  ev.data.fd = fd;
  ev.events = (EPOLLIN | EPOLLPRI | EPOLLONESHOT);
  if ((err = epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &ev)) < 0) {
    log_error("%s", "failed add event via epoll_ctl");
    return err;
  }
  if ((err = pthread_create(&(stats->xen_in_tid), &attr,
&stats_in_tfunc, mod->data)) != 0) {
    log_error("failed to create thread: %s", strerror(err));
    return err;
  }
  if ((err = pthread_detach(stats->xen_in_tid)) != 0) {
    log_error("failed to detach thread: %s", strerror(err));
    return err;
  }


in stats_in_tfunc i'm use this, but looping2 never happening
 for (;;) {
    log_debug("%s", "looping1");
    if ((nr_events = epoll_wait(epfd, xen_events, 10, -1)) < 0) {
      log_error("%s", "failed epoll_wait");
      return NULL;
    }
    log_debug("%s", "looping1");
}



-- 
Vasiliy G Tolstov <v.tolstov@selfip.ru>
Selfip.Ru

Re: use xs_watch from userspace and use epoll instead of select

[Vasiliy G Tolstov]

On Fri, 2011-06-03 at 16:17 +0400, Vasiliy G Tolstov wrote:
> On Fri, 2011-06-03 at 13:13 +0100, Ian Campbell wrote:
> > On Fri, 2011-06-03 at 13:04 +0100, Vasiliy G Tolstov wrote:
> > > Hello. I have some examples of usage xs_watch in userspace that contains
> > > loop with select function usage. If i use epoll, no events returned from
> > > epoll_wait...
> > > Does it matter to use select from fd that used for watch?
> > > epoll fd added with ev.events = (EPOLLIN | EPOLLPRI | EPOLLONESHOT);
> > 
> > I think you need to provide an example of the code which doesn't work.
> > 
> > Ian.
> > 
> 
> This is not work:
>   if ((err = xs_watch(stats->xs, "data/xenmgm", "xenmgm")) != 0) {

Ok. I'm investigate, that xs_read can use relative to root path, for
example i can read domid not provide full path to it
(/local/domain/XX/domid), but if i want use watch, i need provide full
path to wathed node.... This is not clear to me after reading
examples... =)

Thank You.

-- 
Vasiliy G Tolstov <v.tolstov@selfip.ru>
Selfip.Ru

Re: use xs_watch from userspace and use epoll instead of select

[Ian Campbell]

On Fri, 2011-06-03 at 17:07 +0100, Vasiliy G Tolstov wrote:
> On Fri, 2011-06-03 at 16:17 +0400, Vasiliy G Tolstov wrote:
> > On Fri, 2011-06-03 at 13:13 +0100, Ian Campbell wrote:
> > > On Fri, 2011-06-03 at 13:04 +0100, Vasiliy G Tolstov wrote:
> > > > Hello. I have some examples of usage xs_watch in userspace that contains
> > > > loop with select function usage. If i use epoll, no events returned from
> > > > epoll_wait...
> > > > Does it matter to use select from fd that used for watch?
> > > > epoll fd added with ev.events = (EPOLLIN | EPOLLPRI | EPOLLONESHOT);
> > > 
> > > I think you need to provide an example of the code which doesn't work.
> > > 
> > > Ian.
> > > 
> > 
> > This is not work:
> >   if ((err = xs_watch(stats->xs, "data/xenmgm", "xenmgm")) != 0) {
> 
> Ok. I'm investigate, that xs_read can use relative to root path, for
> example i can read domid not provide full path to it
> (/local/domain/XX/domid), but if i want use watch, i need provide full
> path to wathed node.... This is not clear to me after reading
> examples... =)

Which xenstored implementation are you using? In general I would expect
relative watches to work correctly. I looked briefly at the C xenstored
and it appears to (at least be trying to) do the right thing here.

Are you running this code in dom0 or domU?

If it is domU which Xen release is libxenstore you are linking this code
against against from? Which Xen release is your xenstored/dom0 running?

How are you opening your xenstore connection? i.e. which specific
function call, xs_open, xs_daemon_open or xs_domain_open?

Did I understand correctly in your original message that basically the
same code using select() works ok?

In another mail you wrote:
> in stats_in_tfunc i'm use this, but looping2 never happening
>  for (;;) {
>     log_debug("%s", "looping1");
>     if ((nr_events = epoll_wait(epfd, xen_events, 10, -1)) < 0) {
>       log_error("%s", "failed epoll_wait");
>       return NULL;
>     }
>     log_debug("%s", "looping1");
> }

I guess this is a typo in the email rather than the actual code, but
"looping2" never appears in this snippet -- you have "looping1" twice...

Ian.

Re: use xs_watch from userspace and use epoll instead of select

[Vasiliy G Tolstov]

On Mon, 2011-06-06 at 09:41 +0100, Ian Campbell wrote:
> On Fri, 2011-06-03 at 17:07 +0100, Vasiliy G Tolstov wrote:
> > On Fri, 2011-06-03 at 16:17 +0400, Vasiliy G Tolstov wrote:
> > > On Fri, 2011-06-03 at 13:13 +0100, Ian Campbell wrote:
> > > > On Fri, 2011-06-03 at 13:04 +0100, Vasiliy G Tolstov wrote:
> > > > > Hello. I have some examples of usage xs_watch in userspace that contains
> > > > > loop with select function usage. If i use epoll, no events returned from
> > > > > epoll_wait...
> > > > > Does it matter to use select from fd that used for watch?
> > > > > epoll fd added with ev.events = (EPOLLIN | EPOLLPRI | EPOLLONESHOT);
> > > > 
> > > > I think you need to provide an example of the code which doesn't work.
> > > > 
> > > > Ian.
> > > > 
> > > 
> > > This is not work:
> > >   if ((err = xs_watch(stats->xs, "data/xenmgm", "xenmgm")) != 0) {
> > 
> > Ok. I'm investigate, that xs_read can use relative to root path, for
> > example i can read domid not provide full path to it
> > (/local/domain/XX/domid), but if i want use watch, i need provide full
> > path to wathed node.... This is not clear to me after reading
> > examples... =)
> 
> Which xenstored implementation are you using? In general I would expect
> relative watches to work correctly. I looked briefly at the C xenstored
> and it appears to (at least be trying to) do the right thing here.
> 
> Are you running this code in dom0 or domU?
> 
> If it is domU which Xen release is libxenstore you are linking this code
> against against from? Which Xen release is your xenstored/dom0 running?
> 
> How are you opening your xenstore connection? i.e. which specific
> function call, xs_open, xs_daemon_open or xs_domain_open?
> 
> Did I understand correctly in your original message that basically the
> same code using select() works ok?
> 
> In another mail you wrote:
> > in stats_in_tfunc i'm use this, but looping2 never happening
> >  for (;;) {
> >     log_debug("%s", "looping1");
> >     if ((nr_events = epoll_wait(epfd, xen_events, 10, -1)) < 0) {
> >       log_error("%s", "failed epoll_wait");
> >       return NULL;
> >     }
> >     log_debug("%s", "looping1");
> > }
> 
> I guess this is a typo in the email rather than the actual code, but
> "looping2" never appears in this snippet -- you have "looping1" twice...
> 
> Ian.
> 

Code to using from domU. I'm correct all my mistakes and now relative
and full path watches work's fine with epoll and select based looping.
Sorry for wrong panic about it. 

Sorry for small offtop, where i can find example to use grand pages to
transfer for example 1byte from domU to dom) and vice versa?


-- 
Vasiliy G Tolstov <v.tolstov@selfip.ru>
Selfip.Ru

Re: use xs_watch from userspace and use epoll instead of select

[Ian Campbell]

On Mon, 2011-06-06 at 09:58 +0100, Vasiliy G Tolstov wrote:
> Code to using from domU. I'm correct all my mistakes and now relative
> and full path watches work's fine with epoll and select based looping.
> Sorry for wrong panic about it. 

Oh good.

> Sorry for small offtop, where i can find example to use grand pages to
> transfer for example 1byte from domU to dom) and vice versa?

(it's "grant" pages).

Any of the PV drivers would give a decent example for in kernel use of
grant tables to setup a shared memory buffer, but I guess you are asking
for a userspace example? The principals are mostly the same, except you
use xc_* functions instead of the kernel ones.

qemu contains some userspace backends, for example the xenfb backend.
Daniel de Graaf posted some patches to make this use grant tables back
in February/March this year -- you should check the ML archives. There
are also PV disk and net backends in qemu.

I can't think of any userspace frontends offhand.

Once you have a shared page you can use the standard Xen ring.h
datastructures to define your request and response protocol.

What are you actually trying to build?

Ian.

Re: use xs_watch from userspace and use epoll instead of select

[Vasiliy G Tolstov]

On Mon, 2011-06-06 at 10:48 +0100, Ian Campbell wrote:
> On Mon, 2011-06-06 at 09:58 +0100, Vasiliy G Tolstov wrote:
> > Code to using from domU. I'm correct all my mistakes and now relative
> > and full path watches work's fine with epoll and select based looping.
> > Sorry for wrong panic about it. 
> 
> Oh good.
> 
> > Sorry for small offtop, where i can find example to use grand pages to
> > transfer for example 1byte from domU to dom) and vice versa?
> 
> (it's "grant" pages).
> 
> Any of the PV drivers would give a decent example for in kernel use of
> grant tables to setup a shared memory buffer, but I guess you are asking
> for a userspace example? The principals are mostly the same, except you
> use xc_* functions instead of the kernel ones.
> 
> qemu contains some userspace backends, for example the xenfb backend.
> Daniel de Graaf posted some patches to make this use grant tables back
> in February/March this year -- you should check the ML archives. There
> are also PV disk and net backends in qemu.
> 
> I can't think of any userspace frontends offhand.
> 
> Once you have a shared page you can use the standard Xen ring.h
> datastructures to define your request and response protocol.
> 
> What are you actually trying to build?
> 
> Ian.
> 

I'm try to build daemon, that can recieve command via xenstore (xs_watch
for specific path), via /dev/hvcN (serial char interface), now i'm try
to use gntmem mechanism to do this too. If one method not available or
not work or some errors accured - i'm try next. 
/dev/hvcN Not good sometimes, becouse in some domU i have gentty on this
port.
xenstore - has limit in size of message...
In kernel side xen thing more easy to understand. Use it under userspace
need deep googling =).
Ok if i use grant pages mechanism - do i have notify for new message
after it arrives? If i use xs from domU i need libxc under it insted of
libxenstore ?

-- 
Vasiliy G Tolstov <v.tolstov@selfip.ru>
Selfip.Ru

Re: use xs_watch from userspace and use epoll instead of select

[Ian Campbell]

On Mon, 2011-06-06 at 11:05 +0100, Vasiliy G Tolstov wrote:
> On Mon, 2011-06-06 at 10:48 +0100, Ian Campbell wrote:

> > What are you actually trying to build?

> I'm try to build daemon, that can recieve command via xenstore (xs_watch
> for specific path), via /dev/hvcN (serial char interface), now i'm try
> to use gntmem mechanism to do this too. If one method not available or
> not work or some errors accured - i'm try next. 
> /dev/hvcN Not good sometimes, becouse in some domU i have gentty on this
> port.
> xenstore - has limit in size of message...

Interesting.

> In kernel side xen thing more easy to understand. Use it under userspace
> need deep googling =).

I'm afraid so.

Remember that the underlying grant table hypercall's and mechanisms are
the same in both kernel and userspace and the APIs provided reflect that
-- userspace is not all that different.

In userspace you might need (actually I think it's probably required)
to use the gntalloc driver (added to pvops by Daniel de Graff around
2.6.39 I think) to get memory suitable for granting out.

> Ok if i use grant pages mechanism - do i have notify for new message
> after it arrives? If i use xs from domU i need libxc under it insted of
> libxenstore ?

Yes you need to notify, libxc provides an interface to event channels as
well as grant tables.

You will probably need both libxc and libxenstore. You use xenstore to
negotiated the event channel and shared page with your peer and libxc to
create, map, signal them etc.

Ian.

Re: use xs_watch from userspace and use epoll instead of select

[Vasiliy G Tolstov]

On Mon, 2011-06-06 at 12:00 +0100, Ian Campbell wrote:
> On Mon, 2011-06-06 at 11:05 +0100, Vasiliy G Tolstov wrote:
> > On Mon, 2011-06-06 at 10:48 +0100, Ian Campbell wrote:
> 
> > > What are you actually trying to build?
> 
> > I'm try to build daemon, that can recieve command via xenstore (xs_watch
> > for specific path), via /dev/hvcN (serial char interface), now i'm try
> > to use gntmem mechanism to do this too. If one method not available or
> > not work or some errors accured - i'm try next. 
> > /dev/hvcN Not good sometimes, becouse in some domU i have gentty on this
> > port.
> > xenstore - has limit in size of message...
> 
> Interesting.
> 
> > In kernel side xen thing more easy to understand. Use it under userspace
> > need deep googling =).
> 
> I'm afraid so.
> 
> Remember that the underlying grant table hypercall's and mechanisms are
> the same in both kernel and userspace and the APIs provided reflect that
> -- userspace is not all that different.
> 
> In userspace you might need (actually I think it's probably required)
> to use the gntalloc driver (added to pvops by Daniel de Graff around
> 2.6.39 I think) to get memory suitable for granting out.
> 
> > Ok if i use grant pages mechanism - do i have notify for new message
> > after it arrives? If i use xs from domU i need libxc under it insted of
> > libxenstore ?
> 
> Yes you need to notify, libxc provides an interface to event channels as
> well as grant tables.
> 
> You will probably need both libxc and libxenstore. You use xenstore to
> negotiated the event channel and shared page with your peer and libxc to
> create, map, signal them etc.
> 
> Ian.
> 
> 

Thanks for suggestions. I'm try investigate e-mail list with gntdev
patches and write example code..
-- 
Vasiliy G Tolstov <v.tolstov@selfip.ru>
Selfip.Ru

Re: use xs_watch from userspace and use epoll instead of select

[Ian Campbell]

On Mon, 2011-06-06 at 12:18 +0100, Vasiliy G Tolstov wrote:
> On Mon, 2011-06-06 at 12:00 +0100, Ian Campbell wrote:
> > On Mon, 2011-06-06 at 11:05 +0100, Vasiliy G Tolstov wrote:
> > > On Mon, 2011-06-06 at 10:48 +0100, Ian Campbell wrote:
> > 
> > > > What are you actually trying to build?
> > 
> > > I'm try to build daemon, that can recieve command via xenstore (xs_watch
> > > for specific path), via /dev/hvcN (serial char interface), now i'm try
> > > to use gntmem mechanism to do this too. If one method not available or
> > > not work or some errors accured - i'm try next. 
> > > /dev/hvcN Not good sometimes, becouse in some domU i have gentty on this
> > > port.
> > > xenstore - has limit in size of message...
> > 
> > Interesting.
> > 
> > > In kernel side xen thing more easy to understand. Use it under userspace
> > > need deep googling =).
> > 
> > I'm afraid so.
> > 
> > Remember that the underlying grant table hypercall's and mechanisms are
> > the same in both kernel and userspace and the APIs provided reflect that
> > -- userspace is not all that different.
> > 
> > In userspace you might need (actually I think it's probably required)
> > to use the gntalloc driver (added to pvops by Daniel de Graff around
> > 2.6.39 I think) to get memory suitable for granting out.
> > 
> > > Ok if i use grant pages mechanism - do i have notify for new message
> > > after it arrives? If i use xs from domU i need libxc under it insted of
> > > libxenstore ?
> > 
> > Yes you need to notify, libxc provides an interface to event channels as
> > well as grant tables.
> > 
> > You will probably need both libxc and libxenstore. You use xenstore to
> > negotiated the event channel and shared page with your peer and libxc to
> > create, map, signal them etc.
> > 
> > Ian.
> > 
> > 
> 
> Thanks for suggestions. I'm try investigate e-mail list with gntdev
> patches and write example code.

I think it would be great to get some simple examples of doing all these
things from userspace into the tree, or at least posted somewhere.

Ian.