sigopen() vs. /dev/sigtimedwait

sigopen() vs. /dev/sigtimedwait

[Dan Kegel]

So I've been thinking about the sigopen() system call I proposed.
(To recap: sigopen() would let you use read() instead of sigwaitinfo()
 to retrieve lots of realtime signals at one go, AND would
 protect your signal from being swiped by hostile code elsewhere
 in the application, a la Sun's JDK.)

Upon further consideration, maybe I should model it after
/dev/epoll.  That would get rid of nagging questions like
"but read() can't leave holes like sigtimedwait could",
and would be even higher performance than read()
(see graphs at http://www.xmailserver.org/linux-patches/nio-improve.html )

So I'm proposing the following user story:

  // open a fd linked to signal mysignum
  int fd = open("/dev/sigtimedwait", O_RDWR);
  int sigs[1]; sigs[0] = mysignum;
  write(fd, sigs, sizeof(sigs[0]));

  // memory map a result buffer
  struct siginfo_t *map = mmap(NULL, mapsize, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);

  for (;;) {
      // grab recent siginfo_t's
      struct devsiginfo dsi;
      dsi.dsi_nsis = 1000;
      dsi.dsi_sis = NULL;      // NULL means "use map instead of buffer"
      dsi.dsi_timeout = 1;
      int nsis = ioctl(fd, DS_SIGTIMEDWAIT, &dvp);   

      // use 'em.  Some might be completion notifications; some might be readiness notifications.
      for (i=0; i<nsis; i++)
          handle_siginfo(map+i);
  }

Sure, the interface is crap, but it's fast, and at least it doesn't
add any syscalls (the sigopen() proposal required two new syscalls: sigopen()
and timedread()).

Comments?

BTW I'm halfway thru "Understanding the Linux Kernel" and it's
a very good read (modulo some strange lingo, e.g. "cycle" for "loop"
and "table" for "record" or "struct").
So since I only halfway understand the linux kernel, the above proposal
may be half baked.
- Dan

-- 
"I have seen the future, and it licks itself clean." -- Bucky Katt

Re: sigopen() vs. /dev/sigtimedwait

[Petru Paler]

On Fri, Aug 03, 2001 at 06:32:52PM -0700, Dan Kegel wrote:
> So I'm proposing the following user story:
> 
>   // open a fd linked to signal mysignum
>   int fd = open("/dev/sigtimedwait", O_RDWR);
>   int sigs[1]; sigs[0] = mysignum;
>   write(fd, sigs, sizeof(sigs[0]));
> 
>   // memory map a result buffer
>   struct siginfo_t *map = mmap(NULL, mapsize, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
> 
>   for (;;) {
>       // grab recent siginfo_t's
>       struct devsiginfo dsi;
>       dsi.dsi_nsis = 1000;
>       dsi.dsi_sis = NULL;      // NULL means "use map instead of buffer"
>       dsi.dsi_timeout = 1;
>       int nsis = ioctl(fd, DS_SIGTIMEDWAIT, &dvp);   
> 
>       // use 'em.  Some might be completion notifications; some might be readiness notifications.
>       for (i=0; i<nsis; i++)
>           handle_siginfo(map+i);
>   }

And the advantage of this over /dev/epoll would be that you don't have to
explicitly add/remove fd's?

I ask because yesterday I used /dev/epoll in a project and it behaves *very*
well, so I'm wondering what advantages your interface would bring.

> Comments?

How do you handle signal queue overflow? signal-per-fd helps, but you still
have to have the queue as big as the maximum number of fds is...

Petru

Re: sigopen() vs. /dev/sigtimedwait

[Dan Kegel]

Petru Paler wrote:
> 
> On Fri, Aug 03, 2001 at 06:32:52PM -0700, Dan Kegel wrote:
> > So I'm proposing the following user story:
> >
> >   // open a fd linked to signal mysignum
> >   int fd = open("/dev/sigtimedwait", O_RDWR);
> >   int sigs[1]; sigs[0] = mysignum;
> >   write(fd, sigs, sizeof(sigs[0]));
> >
> >   // memory map a result buffer
> >   struct siginfo_t *map = mmap(NULL, mapsize, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
> >
> >   for (;;) {
> >       // grab recent siginfo_t's
> >       struct devsiginfo dsi;
> >       dsi.dsi_nsis = 1000;
> >       dsi.dsi_sis = NULL;      // NULL means "use map instead of buffer"
> >       dsi.dsi_timeout = 1;
> >       int nsis = ioctl(fd, DS_SIGTIMEDWAIT, &dvp);
> >
> >       // use 'em.  Some might be completion notifications; some might be readiness notifications.
> >       for (i=0; i<nsis; i++)
> >           handle_siginfo(map+i);
> >   }
> 
> And the advantage of this over /dev/epoll would be that you don't have to
> explicitly add/remove fd's?

The advantage is that it can be used to collect
completion notifications for aio.  (It can also be
used to collect readiness notification via either
linux's traditional rtsig stuff, or the signal-per-fd stuff,
so this unifies readiness notification and completion notification,
in case you happen to want to use both in the same thread.)
 
> I ask because yesterday I used /dev/epoll in a project and it behaves *very*
> well, so I'm wondering what advantages your interface would bring.

I am a huge fan of /dev/epoll and would like to see it integrated
into the ac series.  /dev/epoll doesn't address the needs of those
who are doing aio, though.
 
> How do you handle signal queue overflow? signal-per-fd helps, but you still
> have to have the queue as big as the maximum number of fds is...

I am not addressing that issue.  However, when doing aio, the
application can simply avoid issuing more than N I/O operations,
where N is comfortably lower than the current size of the signal queue.

When I get around to reading the kernel source finally, maybe I'll
have a look at what the costs of large signal queues are.

- Dan

-- 
"I have seen the future, and it licks itself clean." -- Bucky Katt

Could /dev/epoll deliver aio completion notifications? (was: Re: sigopen() vs. /dev/sigtimedwait)

[Dan Kegel]

Dan Kegel wrote:
> Petru Paler wrote:
> > And the advantage of this over /dev/epoll would be that you don't have to
> > explicitly add/remove fd's?
> 
> The advantage is that it can be used to collect
> completion notifications for aio.  (It can also be
> used to collect readiness notification via either
> linux's traditional rtsig stuff, or the signal-per-fd stuff,
> so this unifies readiness notification and completion notification,
> in case you happen to want to use both in the same thread.)
> 
> > I ask because yesterday I used /dev/epoll in a project and it behaves *very*
> > well, so I'm wondering what advantages your interface would bring.
> 
> I am a huge fan of /dev/epoll and would like to see it integrated
> into the ac series.  /dev/epoll doesn't address the needs of those
> who are doing aio, though.

On the other hand, if /dev/epoll were flexible enough that it could
deliver AIO completion notifications, then /dev/sigtimedwait
would not be needed.  For instance:

// extend bits/poll.h
#define POLLAIO 0x800   // aio completion event; pollfd.fd contains aiocb *

      // open /dev/epoll and set up map as usual
      kdpfd = open("/dev/epoll", O_RDWR);
      char *map = mmap(NULL, mapsize, PROT_READ | PROT_WRITE, MAP_PRIVATE, kdpfd, 0);

      // tell our /dev/epoll fd that we're interested in events on fd diskfd
      struct pollfd pfd;
      pfd.fd = diskfd;
      pfd.events = AIOEVENT;
      pfd.revents = 0;
      write(kdpfd, &pfd, sizeof(pfd));

      // set up an asynchronous read from 'diskfd'
      struct aiocb *r = malloc(sizeof(*r));
      r->aio_filedes = diskfd;
      r->aio_... = ...
      // when read is finished, have it notify the /dev/epoll device 
      // interested in diskfd rather than sending a signal
      r->aio_sigevent.sigev_notify = SIGEV_NONE;
      aio_read(r);
      ...

      // Pick up events
      for (;;) {
          struct devpoll dvp;
          dvp.dp_nfds = 1000;
          dvp.dp_fds = NULL;      // NULL means "use map instead of buffer"
          dvp.dp_timeout = 1;
          int nevents = ioctl(kdpfd, DS_SIGTIMEDWAIT, &dvp);
          struct pollfd *result = map + dvp.result_offset;

          // use 'em.  Some might be aio completion notifications; 
          // some might be traditional poll notifications
          // (and if this is AIX, some might be sysv message queue notifications!)
          for (i=0; i<nevents; i++)
             if (result[i].revents & POLLAIO)
                  handle_aio_completion((struct aiocb *)result[i].fd);
             else 
                  handle_readiness(&result[i]);
      }

Davide, is that along the lines of what you were thinking of
for /dev/epoll and disk files?   (Plain old polling of disk
files doesn't make much sense unless you're just interested in
them growing, I suppose; aio completion notification is what you 
really want.)

- Dan

-- 
"I have seen the future, and it licks itself clean." -- Bucky Katt

Re: Could /dev/epoll deliver aio completion notifications? (was: Re: sigopen() vs. /dev/sigtimedwait)

[Zach Brown]

> On the other hand, if /dev/epoll were flexible enough that it could
> deliver AIO completion notifications, 

As far as I know, Ben LaHaise (bcrl@redhat.com) already has a fine
method conceived for receiving batches of async completion, including an
"async poll".  It should give the sort of behaviour you want and is also
useful for other AIO things, obviously :)

You should really chat with him.

- z

Re: Could /dev/epoll deliver aio completion notifications? (was: Re: sigopen() vs. /dev/sigtimedwait)

[Dan Kegel]

Zach Brown wrote:
> 
> > On the other hand, if /dev/epoll were flexible enough that it could
> > deliver AIO completion notifications,
> 
> As far as I know, Ben LaHaise (bcrl@redhat.com) already has a fine
> method conceived for receiving batches of async completion, including an
> "async poll".  It should give the sort of behaviour you want and is also
> useful for other AIO things, obviously :)
> 
> You should really chat with him.

I suppose it shouldn't suprise me that real kernel hackers like Ben
just work on cool things quietly, and bufoons like me get excited
at the merest thought, and have to broadcast them on l-k.  Sigh.

A little digging finds a few references to Ben's aio work:

http://uwsg.iu.edu/hypermail/linux/kernel/0102.0/0384.html
http://www.kvack.org/~blah/aio/v2.4.5-ac9-bcrl4-aio.diff

His patch creates /dev/aio, among other things, and includes
the wonderful excerpt

diff -urN /md0/kernels/2.4/v2.4.5-ac9/include/asm-i386/errno.h aio-2.4.5-ac9/include/asm-i386/errno.h
--- /md0/kernels/2.4/v2.4.5-ac9/include/asm-i386/errno.h        Mon Feb 26 10:20:14 2001
+++ aio-2.4.5-ac9/include/asm-i386/errno.h      Wed Jun 13 17:08:50 2001
@@ -128,5 +128,6 @@
 
 #define        ENOMEDIUM       123     /* No medium found */
 #define        EMEDIUMTYPE     124     /* Wrong medium type */
+#define        ENOCLUE         125     /* userland programmer induced race condition */

:-)

Given how occupied Ben is with urgent matters like tracking down the
VM suckage, I don't expect he has much time for chatting about this stuff.

OK, guess I'll content myself with working through Ben's (and Davide's)
code and understanding it.  Sorry for all the line noise, folks.
- Dan

-- 
"I have seen the future, and it licks itself clean." -- Bucky Katt

RE: Could /dev/epoll deliver aio completion notifications? (was:

[Davide Libenzi]

On 04-Aug-2001 Dan Kegel wrote:
> Davide, is that along the lines of what you were thinking of
> for /dev/epoll and disk files?   (Plain old polling of disk
> files doesn't make much sense unless you're just interested in
> them growing, I suppose; aio completion notification is what you 
> really want.)

Dan, sorry for the response delay but I was in Vacation Mode ( new CPU execution mode
that will be included in the next x86 generation ).
I trashed my original idea to extend /dev/epoll to other other files coz this will make the
patch way more intrusive. The only easy extension that comes in my mind is pipes.
As soon as I'll finish to read the remaining 4231 messages in my mbox I'll fix /dev/epoll
to get rid of stale events that I ( and Erich Nahum ) noticed in the current implementation.



- Davide