PROBLEM: serial port FIONREAD from realtime thread

PROBLEM: serial port FIONREAD from realtime thread

[Dick Hollenbeck]

Problem Conditions:

1) linux 2.6.11.7, but possibly other kernels too
1) realtime thread
2) serial port open()ed *either* with NON_BLOCKING or not
3) ioctl( FIONREAD ) always returns zero

int ncharin;
ioctl( fd, FIONREAD, &ncharin )

The above lines of code operate incorrectly from a real time thread 
against a normal PC serial port.  The "ncharin" is always set to zero if 
called from a loop.  The work around is to block the calling realtime 
thread with a sleep of some kind.  That workaround is ugly.  We need 
this to work from a realtime thread.

The program below illustrates the problem, but requires two computers.  
On one computer run minicom and connect an RS-232 cross over cable to 
the test box (at COM2, per the test source).  Compile and run the 
program below on the test box.  On the 2nd computer, say a non-test box, 
run minicom at 38400 kbaud to match the baudrate in the test box and 
program below.  Run the test program as root so it can escalate the 
priority to realtime.  Minicom should be used to send chars to the test 
program during the 20 seconds that the test program runs.

Test program will only run for 20 seconds.  While it's running, you 
should send characters from minicom to the test box and you will see the 
buffered count of characters increment by 1.  That is how it is supposed 
to work.  Then comment out the usleep() call and recompile and do the 
same test again.  Here you see the test program does not operate 
properly.  The buffered character count remains at 0.

What causes this?   What is the usleep() enabling?

-----------------------------------

#include <unistd.h>
#include <fcntl.h>
#include <termios.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <linux/serial.h>
#include <sched.h>
#include <time.h>

int fd;

int openPort();
void setRealtimePriority();


int main( int argc, char** argv )
{
   int         ncharin;
   int         last;
   time_t      start;
     if( openPort() != 0 )
   {
       fprintf( stderr, "unable to open port\n" );
       exit(1);
   }
     setRealtimePriority();
     // poll the FIONREAD and print the number of chars in the recv buffers

   start = time(NULL);  
   // run only for 20 seconds, since realtime priority steals cpu from 
consoles.
   while( time(NULL) - start < 20 )
   {
       if( -1 == ioctl( fd, FIONREAD, &ncharin ) )
       {
           printf( "error from ioctl\n" );
           ncharin = 0;
       }

       // if this line is commented out, then ioctl FIONREAD does not work.
       usleep(1);
         if( last != ncharin )
       {
           // this should show a monotonically increasing number of 
characters.
           printf(" %d", ncharin );
                 fflush(stdout);
                     last = ncharin;
       }
   }
}


int openPort()
{
   struct termios     t;

   /*  The O_NDELAY flag tells UNIX that this program doesn't care what
       state the DCD signal line is in - whether the other end of the port
       is up and running. If you do not specify this flag, your process 
will
       be put to sleep until the DCD signal line is the space voltage.
   */
   fd = open( "/dev/ttyS1", O_RDWR
                           | O_NDELAY
                           | O_NOCTTY );

   if( fd == -1 )
       return -1;

   memset( &t, 0, sizeof(t) );

   cfsetispeed( &t, B38400 );
   cfsetospeed( &t, B38400 );

   // Enable the receiver and set local mode...
   //  CLOCAL  : local connection, no modem contol
   //  CREAD   : enable receiving characters
   t.c_cflag |= (CLOCAL | CREAD);

   t.c_cflag &= ~CSTOPB;

   t.c_cflag &= ~CSIZE;    // Mask the character size bits
   t.c_cflag |= CS8;       // Select 8 data bits

   t.c_cflag &= ~PARENB;
   t.c_cflag &= ~CSTOPB;

   // Raw input is unprocessed.
   // Input characters are passed through exactly as they are received,
   // when they are received. Generally you'll deselect the ICANON, ECHO,
   // ECHOE, and ISIG options when using raw input:
   t.c_lflag &= ~(ICANON | ECHO | ECHOE | ISIG);

   // Raw output is selected by resetting the OPOST option in the  
c_oflag member:
   t.c_oflag &= ~OPOST;

   t.c_cc[VMIN]  = 1;
   t.c_cc[VTIME] = 0;

   // Set the new options for the port...
   tcsetattr( fd, TCSANOW, &t );

   return 0;
}



void setRealtimePriority()
{
   int ec;

   printf("setRealtimePriority()\n");
     struct sched_param     p;

   memset( &p, 0, sizeof(p) );

   p.sched_priority = 2;

   ec = sched_setscheduler( 0, SCHED_RR, &p );
   if( ec != 0 )
   {
       fprintf(stderr, "pthread_setschedparam ec=%d, %s\n", ec, 
strerror(ec) );
       fflush(stderr);
       exit(2);
   }
}

Re: PROBLEM: serial port FIONREAD from realtime thread

[Paul Fulghum]

Dick Hollenbeck wrote:
> Problem Conditions:
> 
> 1) linux 2.6.11.7, but possibly other kernels too
> 1) realtime thread
> 2) serial port open()ed *either* with NON_BLOCKING or not
> 3) ioctl( FIONREAD ) always returns zero
> ...
> What causes this?   What is the usleep() enabling?

The tty flip buffering uses a workqueue
to push received data to the line discipline.

Your polling loop at raised priority
is probably preventing the workqueue
from being processed by the events
kernel thread.

When you call usleep() your process yields
so receive data can be processed.

Have you tried (just for testing)
lowering your process priority below that
of the events kernel thread?

-- 
Paul Fulghum
Microgate Systems, Ltd.

Re: PROBLEM: serial port FIONREAD from realtime thread

[Dick Hollenbeck]

Paul Fulghum wrote:

> Dick Hollenbeck wrote:
>
>> Problem Conditions:
>>
>> 1) linux 2.6.11.7, but possibly other kernels too
>> 1) realtime thread
>> 2) serial port open()ed *either* with NON_BLOCKING or not
>> 3) ioctl( FIONREAD ) always returns zero
>> ...
>> What causes this?   What is the usleep() enabling?
>
>
> The tty flip buffering uses a workqueue
> to push received data to the line discipline.
>
> Your polling loop at raised priority
> is probably preventing the workqueue
> from being processed by the events
> kernel thread.
>
> When you call usleep() your process yields
> so receive data can be processed.
>
> Have you tried (just for testing)
> lowering your process priority below that
> of the events kernel thread?
>
Thanks Paul.  This sounds right.  No, I have not dialed in at such a 
fine granularity.  Obviously at a non realtime priority things work OK, 
and obviously I need to run at a realtime priority.  There may be some 
room to fit in between.

1) Can you point me at the source file(s) where I can explore this 
further? (source file the workqueue written to).

2) How do I determine the priority of the events kernel thread (where is 
it created, and/or priority set).


Thanks much.

Re: PROBLEM: serial port FIONREAD from realtime thread

[Paul Fulghum]

Dick Hollenbeck wrote:
> 1) Can you point me at the source file(s) where I can explore this 
> further? (source file the workqueue written to).

drivers/char/tty_io.c has the flip buffer code

The function tty_flip_buffer_push() is called by
the hardware driver when data has been added to
the flip buffer. This function calls flush_to_ldisc()
either directly (if low_latency is set) or indirectly
through the workqueue.

There is a problem in setting low_latency if
tty_flip_buffer_push() is called in interrupt context,
which many drivers do. You can try setting this and
seeing if your machine crashes and burns.

This code is a moving target at this time.
Alan Cox is working on bringing
a small measure of sanity to the code.

> 2) How do I determine the priority of the events kernel thread (where is 
> it created, and/or priority set).

You should be able to look at the output of
'top' or 'ps' for the 'events/0' entry.

My machine shows a priority of 10.

-- 
Paul Fulghum
Microgate Systems, Ltd.

PROBLEM: serial port FIONREAD from realtime thread

[Dick Hollenbeck]

Problem Conditions:

1) linux 2.6.11.7, but is probably otherplaces too
1) realtime thread
2) serial port open()ed *either* with NON_BLOCKING or not
3) ioctl( FIONREAD ) always returns zero

int ncharin;
ioctl( fd, FIONREAD, &ncharin )

The above lines of code operate incorrectly from a real time thread on a 
normal PC serial port.  They ncharin is always set to zero if called 
from a loop.  The work around is to block the calling realtime thread 
with a sleep of some kind.  Said workaround is ugly.  Need this to work 
from a realtime thread.

The program below illustrates the problem, but requires two computers.  
On one computer run minicom and connect an RS-232 cross over cable to 
the test box (at COM2, per the test source).  Compiler and run the 
program below on the test box.  Set minicom to 38400 kbaud to match the 
baudrate in the program below.  Compile test program.  Run the test 
program as root so it can escalate the priority to realtime.

It will only run for 20 seconds, while it's running, you should send 
characters from minicom to the test box and you will see the buffered 
count of characters increment by 1.  That is how it is supposed to 
work.  Then comment out the usleep() call and recompile and do the same 
test again.  Here you see the test program does not operate properly.  
The buffered character count remains at 0.     What causes this?  Where 
do I patch?

-----------------------------------

#include <unistd.h>
#include <fcntl.h>
#include <termios.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <linux/serial.h>
#include <sched.h>
#include <time.h>

int fd;

int openPort();
void setRealtimePriority();


int main( int argc, char** argv )
{
    int         ncharin;
    int         last;
    time_t      start;
   
    if( openPort() != 0 )
    {
        fprintf( stderr, "unable to open port\n" );
        exit(1);
    }
   
    setRealtimePriority();
   
    // poll the FIONREAD and print the number of chars in the recv buffers

    start = time(NULL);   

    // run only for 20 seconds, since realtime priority steals cpu from 
consoles.
    while( time(NULL) - start < 20 )
    {
        if( -1 == ioctl( fd, FIONREAD, &ncharin ) )
        {
            printf( "error from ioctl\n" );
            ncharin = 0;
        }

        // if this line is commented out, then ioctl FIONREAD does not work.
        usleep(1);
   
        if( last != ncharin )
        {
            // this should show a monotonically increasing number of 
characters.
            printf(" %d", ncharin );
       
            fflush(stdout);
           
            last = ncharin;
        }
    }
}


int openPort()
{
    struct termios     t;

    /*  The O_NDELAY flag tells UNIX that this program doesn't care what
        state the DCD signal line is in - whether the other end of the port
        is up and running. If you do not specify this flag, your process 
will
        be put to sleep until the DCD signal line is the space voltage.
    */
    fd = open( "/dev/ttyS1", O_RDWR
                            | O_NDELAY
                            | O_NOCTTY );

    if( fd == -1 )
        return -1;

    memset( &t, 0, sizeof(t) );

    cfsetispeed( &t, B38400 );
    cfsetospeed( &t, B38400 );

    // Enable the receiver and set local mode...
    //  CLOCAL  : local connection, no modem contol
    //  CREAD   : enable receiving characters
    t.c_cflag |= (CLOCAL | CREAD);

    t.c_cflag &= ~CSTOPB;

    t.c_cflag &= ~CSIZE;    // Mask the character size bits
    t.c_cflag |= CS8;       // Select 8 data bits

    t.c_cflag &= ~PARENB;
    t.c_cflag &= ~CSTOPB;

    // Raw input is unprocessed.
    // Input characters are passed through exactly as they are received,
    // when they are received. Generally you'll deselect the ICANON, ECHO,
    // ECHOE, and ISIG options when using raw input:
    t.c_lflag &= ~(ICANON | ECHO | ECHOE | ISIG);

    // Raw output is selected by resetting the OPOST option in the  
c_oflag member:
    t.c_oflag &= ~OPOST;

    t.c_cc[VMIN]  = 1;
    t.c_cc[VTIME] = 0;

    // Set the new options for the port...
    tcsetattr( fd, TCSANOW, &t );

    return 0;
}



void setRealtimePriority()
{
    int ec;

    printf("setRealtimePriority()\n");
   
    struct sched_param     p;

    memset( &p, 0, sizeof(p) );

    p.sched_priority = 2;

    ec = sched_setscheduler( 0, SCHED_RR, &p );
    if( ec != 0 )
    {
        fprintf(stderr, "pthread_setschedparam ec=%d, %s\n", ec, 
strerror(ec) );
        fflush(stderr);
        exit(2);
    }
}