About I/O callbacks ...

About I/O callbacks ...

[Davide Libenzi]

I was just thinking to implement I/O callbacks inside the kernel and test which
kind of performance could result compared to a select()/poll() implementation.
I prefer IO callbacks against async-io coz in this way is more direct to
implement an I/O driven state machine + coroutines.
This is a first draft :



#define ASCB_INFO_NPARAMS 4

struct ascb_info {
        unsigned long param[ASCB_INFO_NPARAMS];
};

struct iocb {
        struct iocb * next;
        struct task_struct * task;
        unsigned long event;
        int (*iocb_func)(struct ascb_info *, void *);
        void * data;
};

struct ascb {
        struct ascb * next;
        int (*ascb_func)(struct ascb_info *, void *);
        void * data;
        struct ascb_info info;
};





struct task_struct {
        ...
        struct ascb * ascb_list;
        spinlock_t ascb_list_lock;
        ...
};


struct file {
        ...
        struct iocb * iocb_list;
        spinlock_t iocb_list_lock;
        ...
};




The user call some user-space api to add the callback to the fd and this will
result ( inside the kernel ) to a call to :



int iocb_file_add(struct file * file,
                int (*iocb_func)(struct ascb_info *, void *), void * data,
                unsigned long event) {

/*
 *  Add the callback to the file list with task = current
 */


}



This is used to add callbacks to the task's list :

int ascb_task_add(struct task_struct * task,
                int (*ascb_func)(struct ascb_info *, void *),
                void * data, struct ascb_info * info) {

/*
 *  Add the callback to the task list
 */

}




Low level I/O layers will call this to dispatch I/O events :


int iocb_file_dispatch(struct file * file, unsigned long event) {

/*
 *  Scan the iocb_list and ( if event match ) call ascb_task_add()
 *  and remove ( y/n ? ) the callback from iocb_list
 */


}




In entry.S we'll have a call like do_signal() that will build the frame for a
callback call ( like do_signal() ) and will remove the entry from the list.
My first implementation should address only sockets but once the concept of
async callbacks is inside the task_struct this could be gradually extended to
std files and even used as an extension of signals.

Comments ?





- Davide

Re: About I/O callbacks ...

[Andi Kleen]

Davide Libenzi <davidel@xmailserver.org> writes:

> I was just thinking to implement I/O callbacks inside the kernel and test which
> kind of performance could result compared to a select()/poll() implementation.
> I prefer IO callbacks against async-io coz in this way is more direct to
> implement an I/O driven state machine + coroutines.
> This is a first draft :

They already exist since several years in Linux. 

It is called queued SIGIO; when you set a realtime signal
using F_SETSIG for a fd and enable async IO. The callback is a signal
handler or alternatively an event processing loop using sigwaitinfo*(). 
The necessary information like which fd got an event is passed in the 
siginfo_t. 

With that mechanism you could easily implement your API in user space.

Currently it doesn't work for block devices of course; there is no real
kernel async block IO in linux kernel; but fine for networking and some
other devices.

It works fine in most cases. Most annoying design bug is that it can be
only sent to the current thread for non root processes (that is caused
by an older overzealous security fix); unfortunately fixing it is 2.5
material because it requires storing the sender credentials in the socket
for this case. Also some people don't like the behaviour on signal queue 
overflow. Signal delivery is currently not the most optimized path in 
Linux, but I doubt that you could get it much faster because essentially your 
callbacks have to do the same work as signals. Also you would need to solve
the exact same problems, as efficient overload handling.

The SSE2 changes in 2.4 hurt somewhat because it now saves a big SSE
context.  For one application I implemented a SA_NOFP flag which brings
the signal deliver to near 2.2 speed again if you don't need FP in the signal 
handler.

--- arch/i386/kernel/i387.c-NOFP	Sat Mar 10 01:24:00 2001
+++ arch/i386/kernel/i387.c	Mon May  7 14:59:18 2001
@@ -323,11 +323,6 @@
 	if ( !current->used_math )
 		return 0;
 
-	/* This will cause a "finit" to be triggered by the next
-	 * attempted FPU operation by the 'current' process.
-	 */
-	current->used_math = 0;
-
 	if ( HAVE_HWFP ) {
 		if ( cpu_has_fxsr ) {
 			return save_i387_fxsave( buf );
@@ -335,6 +330,11 @@
 			return save_i387_fsave( buf );
 		}
 	} else {
+		/* This will cause a "finit" to be triggered by the next
+		 * attempted FPU operation by the 'current' process.
+		 */
+		current->used_math = 0;
+       
 		return save_i387_soft( &current->thread.i387.soft, buf );
 	}
 }
--- arch/i386/kernel/signal.c-NOFP	Sun Feb 18 15:00:22 2001
+++ arch/i386/kernel/signal.c	Mon May  7 14:55:42 2001
@@ -316,11 +316,11 @@
 
 static int
 setup_sigcontext(struct sigcontext *sc, struct _fpstate *fpstate,
-		 struct pt_regs *regs, unsigned long mask)
+		 struct pt_regs *regs, unsigned long mask, int fp)
 {
-	int tmp, err = 0;
+	int err = 0;
+	int tmp;
 
-	tmp = 0;
 	__asm__("movl %%gs,%0" : "=r"(tmp): "0"(tmp));
 	err |= __put_user(tmp, (unsigned int *)&sc->gs);
 	__asm__("movl %%fs,%0" : "=r"(tmp): "0"(tmp));
@@ -344,11 +344,12 @@
 	err |= __put_user(regs->esp, &sc->esp_at_signal);
 	err |= __put_user(regs->xss, (unsigned int *)&sc->ss);
 
-	tmp = save_i387(fpstate);
-	if (tmp < 0)
+	if (fp)
+	  fp = save_i387(fpstate);
+	if (fp < 0)
 	  err = 1;
 	else
-	  err |= __put_user(tmp ? fpstate : NULL, &sc->fpstate);
+	  err |= __put_user(fp ? fpstate : NULL, &sc->fpstate);
 
 	/* non-iBCS2 extensions.. */
 	err |= __put_user(mask, &sc->oldmask);
@@ -404,7 +405,8 @@
 	if (err)
 		goto give_sigsegv;
 
-	err |= setup_sigcontext(&frame->sc, &frame->fpstate, regs, set->sig[0]);
+	err |= setup_sigcontext(&frame->sc, &frame->fpstate, regs, set->sig[0], 
+				!!(ka->sa.sa_flags&SA_NOFP));
 	if (err)
 		goto give_sigsegv;
 
@@ -485,7 +487,7 @@
 			  &frame->uc.uc_stack.ss_flags);
 	err |= __put_user(current->sas_ss_size, &frame->uc.uc_stack.ss_size);
 	err |= setup_sigcontext(&frame->uc.uc_mcontext, &frame->fpstate,
-			        regs, set->sig[0]);
+			        regs, set->sig[0], !!(ka->sa.sa_flags&SA_NOFP));
 	err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set));
 	if (err)
 		goto give_sigsegv;
--- include/asm-i386/signal.h-NOFP	Wed Sep 13 23:29:16 2000
+++ include/asm-i386/signal.h	Mon May  7 14:41:37 2001
@@ -80,6 +80,7 @@
  * SA_RESETHAND clears the handler when the signal is delivered.
  * SA_NOCLDWAIT flag on SIGCHLD to inhibit zombies.
  * SA_NODEFER prevents the current signal from being masked in the handler.
+ * SA_NOFP    Don't save FP state.	
  *
  * SA_ONESHOT and SA_NOMASK are the historical Linux names for the Single
  * Unix names RESETHAND and NODEFER respectively.
@@ -97,6 +98,7 @@
 #define SA_INTERRUPT	0x20000000 /* dummy -- ignored */
 
 #define SA_RESTORER	0x04000000
+#define SA_NOFP		0x02000000
 
 /* 
  * sigaltstack controls




-Andi

Re: About I/O callbacks ...

[Davide Libenzi]

On 22-Jun-2001 Andi Kleen wrote:
> Davide Libenzi <davidel@xmailserver.org> writes:
> 
>> I was just thinking to implement I/O callbacks inside the kernel and test
>> which
>> kind of performance could result compared to a select()/poll()
>> implementation.
>> I prefer IO callbacks against async-io coz in this way is more direct to
>> implement an I/O driven state machine + coroutines.
>> This is a first draft :
> 
> They already exist since several years in Linux. 
> 
> It is called queued SIGIO; when you set a realtime signal
> using F_SETSIG for a fd and enable async IO. The callback is a signal
> handler or alternatively an event processing loop using sigwaitinfo*(). 
> The necessary information like which fd got an event is passed in the 
> siginfo_t. 

I know about rt signals and SIGIO :) but I can't see how You can queue signals :

current->sig->action[..]

The action field is an array so if more than one I/O notification is fired
before the SIGIO is delivered, You'll deliver only the last one.
Am I missing something ?




- Davide

Re: About I/O callbacks ...

[Andi Kleen]

On Fri, Jun 22, 2001 at 08:55:00AM -0700, Davide Libenzi wrote:
> I know about rt signals and SIGIO :) but I can't see how You can queue signals :
> 
> current->sig->action[..]
> 
> The action field is an array so if more than one I/O notification is fired
> before the SIGIO is delivered, You'll deliver only the last one.
> Am I missing something ?

Yes. Realtime signals (>=SIGRTMIN) get queued in current->pending.

-Andi

Re: About I/O callbacks ...

[Davide Libenzi]

On 22-Jun-2001 Andi Kleen wrote:
> On Fri, Jun 22, 2001 at 08:55:00AM -0700, Davide Libenzi wrote:
>> I know about rt signals and SIGIO :) but I can't see how You can queue
>> signals :
>> 
>> current->sig->action[..]
>> 
>> The action field is an array so if more than one I/O notification is fired
>> before the SIGIO is delivered, You'll deliver only the last one.
>> Am I missing something ?
> 
> Yes. Realtime signals (>=SIGRTMIN) get queued in current->pending.

Damn right.



- Davide

Re: About I/O callbacks ...

[Davide Libenzi]

On 22-Jun-2001 Andi Kleen wrote:
> On Fri, Jun 22, 2001 at 08:55:00AM -0700, Davide Libenzi wrote:
>> I know about rt signals and SIGIO :) but I can't see how You can queue
>> signals :
>> 
>> current->sig->action[..]
>> 
>> The action field is an array so if more than one I/O notification is fired
>> before the SIGIO is delivered, You'll deliver only the last one.
>> Am I missing something ?
> 
> Yes. Realtime signals (>=SIGRTMIN) get queued in current->pending.

An issue with rt signal is the limited number of this resource that
will force you to have a mapping  fd ==> coroutine  but this should not be a
big problem using  hash( fd ) = fd





- Davide