[Xenomai] Loose determinism when reading pcap and sending as raw ethernet packets

[Xenomai] Loose determinism when reading pcap and sending as raw ethernet packets

[Umair Ali]

Hi there,

I am reading the pcap file using the following technique:
"
mlockall(MCL_CURRENT|MCL_FUTURE);
fd = open (file.pcap, O_RDONLY);
status = fstat (fd, & s);
size = s.st_size;
 /* Memory-map the file. */
 mapped = mmap (0, size, PROT_READ, MAP_PRIVATE, fd, 0);"


Then i have the pcap file in mapped variable in the form of chracters (hex numbers). I read packet by packet and send them as raw packet over the network with following code:
"struct timespec ts;
ts.tv_sec = 0;
ts.tv_nsec = 5000;
j=24;

 while (j < size){
         i = (unsigned char) mapped[j + 8];
         sendto(sock, &mapped[j+16], i, 0,(struct sockaddr *)&peer, sizeof(peer));
         nanosleep(&ts, NULL);
          j = j+i+16;
}
"

When i run the program and receive the packets on the receiver then i have observed that behaviour of sending raw packets is not constant. Sometimes nanosleep finish little late or quickly.
The slackspot output file is attached for reference. Please guide that what am i doing wrong.


Thanks & Regards
Ali

-------------- next part --------------
A non-text attachment was scrubbed...
Name: slackspot_output
Type: application/octet-stream
Size: 8999 bytes
Desc: slackspot_output
URL: <http://xenomai.org/pipermail/xenomai/attachments/20151208/3763c1b0/attachment.obj>

Re: [Xenomai] Loose determinism when reading pcap and sending as raw ethernet packets

[Gilles Chanteperdrix]

On Tue, Dec 08, 2015 at 05:25:21PM +0000, Umair Ali wrote:
> Hi there,
> 
> I am reading the pcap file using the following technique:
> "
> mlockall(MCL_CURRENT|MCL_FUTURE);
> fd = open (file.pcap, O_RDONLY);
> status = fstat (fd, & s);
> size = s.st_size;
>  /* Memory-map the file. */
>  mapped = mmap (0, size, PROT_READ, MAP_PRIVATE, fd, 0);"
> 
> 
> Then i have the pcap file in mapped variable in the form of chracters (hex numbers). I read packet by packet and send them as raw packet over the network with following code:
> "struct timespec ts;
> ts.tv_sec = 0;
> ts.tv_nsec = 5000;
> j=24;
> 
>  while (j < size){
>          i = (unsigned char) mapped[j + 8];
>          sendto(sock, &mapped[j+16], i, 0,(struct sockaddr *)&peer, sizeof(peer));
>          nanosleep(&ts, NULL);
>           j = j+i+16;
> }
> "
> 
> When i run the program and receive the packets on the receiver
> then i have observed that behaviour of sending raw packets is not
> constant. Sometimes nanosleep finish little late or quickly.

There are several problems with this program:
- for a program to run in primary mode only, pthread_setschedparam
should be called with a priority higher than 1 for the SCHED_FIFO
(or SCHED_RR) policy;
- when you call a function which returns a value, like send or
nanosleep, you should always check the return value for errors;
- if you want to send a message on average every 5us then
nanosleep(5us) is not what you should be using, you should be using
clock_nanosleep with an absolute date, or the easier but less
portable timerfd;
- what you observe is the jitter, you can measure the timer
interrupt jitter with the "latency" test, but I bet that maybe the
latency may be higher than 5us;
- if you really really need to reduce the jitter, then using kernel
timers instead of sleeping in a user-space thread is recommended,
see:
http://letsmakerobots.com/node/28812
http://letsmakerobots.com/node/32347

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] Loose determinism when reading pcap and sending as raw ethernet packets

[Umair Ali]

Hi Gilles,

Thanks for understanding my problem.

>- for a program to run in primary mode only, pthread_setschedparam
should be called with a priority higher than 1 for the SCHED_FIFO
(or SCHED_RR) policy;

I have been doing the same already but i did not copy that part of code in the email.

>- when you call a function which returns a value, like send or
nanosleep, you should always check the return value for errors;
- if you want to send a message on average every 5us then
nanosleep(5us) is not what you should be using, you should be using
clock_nanosleep with an absolute date, or the easier but less
portable timerfd;

I have used the clock_nanosleep with absolute time, but i could not find the absolute date. With clock_nanosleep i have found the same problem as described in the last email.  I have looked the file timerfd.h. What i have understand is that by using timerfd i will create the dedicated timer with unique ID and then use the ID to get the time by using timerfd_gettime(). The time is added with amount of delay and passed to clock_nanosleep() function to achieve the required sleep time period. Am i doing it right with timerfd?

>  what you observe is the jitter, you can measure the timer
interrupt jitter with the "latency" test, but I bet that maybe the
latency may be higher than 5us;
- if you really really need to reduce the jitter, then using kernel
timers instead of sleeping in a user-space thread is recommended,

Yes, you are right that what i have observed is the jitter. I have gone through the links which you have provided me in the last email.  I am building my application using POSIX skin. In the links, they have used the rtdm_timer_start(). Now my question is that can i use the rtdm functions in the application compiling with posix skin, if yes then how. Do i have to change the skin for comilation in order to use the rtdm functions. Please refer me some guide for using the Xenomai Kernel  timers with posix skin in order to avoid the jitter. Last thing do i need to made a kernel module in order to use the kernel timers. I am confused. Please guide me.

Thanks & BR
Ali
________________________________________
From: Gilles Chanteperdrix [gilles.chanteperdrix@xenomai.org]
Sent: Tuesday, December 08, 2015 10:13 PM
To: Umair Ali
Cc: xenomai@xenomai.org
Subject: Re: [Xenomai] Loose determinism when reading pcap and sending as raw ethernet packets

On Tue, Dec 08, 2015 at 05:25:21PM +0000, Umair Ali wrote:
> Hi there,
>
> I am reading the pcap file using the following technique:
> "
> mlockall(MCL_CURRENT|MCL_FUTURE);
> fd = open (file.pcap, O_RDONLY);
> status = fstat (fd, & s);
> size = s.st_size;
>  /* Memory-map the file. */
>  mapped = mmap (0, size, PROT_READ, MAP_PRIVATE, fd, 0);"
>
>
> Then i have the pcap file in mapped variable in the form of chracters (hex numbers). I read packet by packet and send them as raw packet over the network with following code:
> "struct timespec ts;
> ts.tv_sec = 0;
> ts.tv_nsec = 5000;
> j=24;
>
>  while (j < size){
>          i = (unsigned char) mapped[j + 8];
>          sendto(sock, &mapped[j+16], i, 0,(struct sockaddr *)&peer, sizeof(peer));
>          nanosleep(&ts, NULL);
>           j = j+i+16;
> }
> "
>
> When i run the program and receive the packets on the receiver
> then i have observed that behaviour of sending raw packets is not
> constant. Sometimes nanosleep finish little late or quickly.

There are several problems with this program:
- for a program to run in primary mode only, pthread_setschedparam
should be called with a priority higher than 1 for the SCHED_FIFO
(or SCHED_RR) policy;
- when you call a function which returns a value, like send or
nanosleep, you should always check the return value for errors;
- if you want to send a message on average every 5us then
nanosleep(5us) is not what you should be using, you should be using
clock_nanosleep with an absolute date, or the easier but less
portable timerfd;
- what you observe is the jitter, you can measure the timer
interrupt jitter with the "latency" test, but I bet that maybe the
latency may be higher than 5us;
- if you really really need to reduce the jitter, then using kernel
timers instead of sleeping in a user-space thread is recommended,
see:
http://letsmakerobots.com/node/28812
http://letsmakerobots.com/node/32347

--
                                            Gilles.
https://click-hack.org

Re: [Xenomai] Loose determinism when reading pcap and sending as raw ethernet packets

[Gilles Chanteperdrix]

On Thu, Dec 10, 2015 at 01:20:47PM +0000, Umair Ali wrote:
> Hi Gilles,

Hi,

> 
> Thanks for understanding my problem.
> 
> >- for a program to run in primary mode only, pthread_setschedparam
> should be called with a priority higher than 1 for the SCHED_FIFO
> (or SCHED_RR) policy;
> 
> I have been doing the same already but i did not copy that part of code in the email.
> 
> >- when you call a function which returns a value, like send or
> nanosleep, you should always check the return value for errors;
> - if you want to send a message on average every 5us then
> nanosleep(5us) is not what you should be using, you should be using
> clock_nanosleep with an absolute date, or the easier but less
> portable timerfd;
> 
> I have used the clock_nanosleep with absolute time, but i could
> not find the absolute date. With clock_nanosleep i have found the
> same problem as described in the last email.

You compute the next absolute date. But the point is to add 5us to
the absolute date, instead of using now + 5us, so that the packets
are really sent every 5us on average. This should not solve the
problem you have, only ensure that the messages are sent every 5us
on average.

> I have looked the
> file timerfd.h. What i have understand is that by using timerfd i
> will create the dedicated timer with unique ID and then use the ID
> to get the time by using timerfd_gettime(). The time is added with
> amount of delay and passed to clock_nanosleep() function to
> achieve the required sleep time period. Am i doing it right with
> timerfd?

With timerfd, you would set up the timer with timerfd_settime() to
tick every 5us, then the read() system call to wait for the timer
next expiration.

> 
> >  what you observe is the jitter, you can measure the timer
> interrupt jitter with the "latency" test, but I bet that maybe the
> latency may be higher than 5us;
> - if you really really need to reduce the jitter, then using kernel
> timers instead of sleeping in a user-space thread is recommended,
> 
> Yes, you are right that what i have observed is the jitter. I have
> gone through the links which you have provided me in the last
> email. I am building my application using POSIX skin. In the
> links, they have used the rtdm_timer_start(). Now my question is
> that can i use the rtdm functions in the application compiling
> with posix skin, if yes then how. Do i have to change the skin for
> comilation in order to use the rtdm functions. Please refer me
> some guide for using the Xenomai Kernel timers with posix skin in
> order to avoid the jitter. Last thing do i need to made a kernel
> module in order to use the kernel timers. I am confused. Please
> guide me.

If you want to use kernel-space, you need to write a driver, running
in kernel spaces, and drivers use the RTDM API. The POSIX API is for
the application.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] Loose determinism when reading pcap and sending as raw ethernet packets

[Umair Ali]

Hello,

> >
> >  what you observe is the jitter, you can measure the timer
> interrupt jitter with the "latency" test, but I bet that maybe the
> latency may be higher than 5us;
> - if you really really need to reduce the jitter, then using kernel
> timers instead of sleeping in a user-space thread is recommended,
>
> Yes, you are right that what i have observed is the jitter. I have
> gone through the links which you have provided me in the last
> email. I am building my application using POSIX skin. In the
> links, they have used the rtdm_timer_start(). Now my question is
> that can i use the rtdm functions in the application compiling
> with posix skin, if yes then how. Do i have to change the skin for
> comilation in order to use the rtdm functions. Please refer me
> some guide for using the Xenomai Kernel timers with posix skin in
> order to avoid the jitter. Last thing do i need to made a kernel
> module in order to use the kernel timers. I am confused. Please
> guide me.

>If you want to use kernel-space, you need to write a driver, running
in kernel spaces, and drivers use the RTDM API. The POSIX API is for
the application.

As i need to send raw Ethernet packets therefore i am using  rtnet driver for this purpose. Do you mean that i have to write my own driver for ethernet card.  Or should i have to write an rtdm driver that uses rtnet drivers for sending the raw ethernet. Or is there a way i can change the rtnet drivers according to my need. 

Thanks & BR
ali
________________________________________
From: Gilles Chanteperdrix [gilles.chanteperdrix@xenomai.org]
Sent: Thursday, December 10, 2015 5:01 PM
To: Umair Ali
Cc: xenomai@xenomai.org
Subject: Re: [Xenomai] Loose determinism when reading pcap and sending as raw ethernet packets

On Thu, Dec 10, 2015 at 01:20:47PM +0000, Umair Ali wrote:
> Hi Gilles,

Hi,

>
> Thanks for understanding my problem.
>
> >- for a program to run in primary mode only, pthread_setschedparam
> should be called with a priority higher than 1 for the SCHED_FIFO
> (or SCHED_RR) policy;
>
> I have been doing the same already but i did not copy that part of code in the email.
>
> >- when you call a function which returns a value, like send or
> nanosleep, you should always check the return value for errors;
> - if you want to send a message on average every 5us then
> nanosleep(5us) is not what you should be using, you should be using
> clock_nanosleep with an absolute date, or the easier but less
> portable timerfd;
>
> I have used the clock_nanosleep with absolute time, but i could
> not find the absolute date. With clock_nanosleep i have found the
> same problem as described in the last email.

You compute the next absolute date. But the point is to add 5us to
the absolute date, instead of using now + 5us, so that the packets
are really sent every 5us on average. This should not solve the
problem you have, only ensure that the messages are sent every 5us
on average.

> I have looked the
> file timerfd.h. What i have understand is that by using timerfd i
> will create the dedicated timer with unique ID and then use the ID
> to get the time by using timerfd_gettime(). The time is added with
> amount of delay and passed to clock_nanosleep() function to
> achieve the required sleep time period. Am i doing it right with
> timerfd?

With timerfd, you would set up the timer with timerfd_settime() to
tick every 5us, then the read() system call to wait for the timer
next expiration.

>
> >  what you observe is the jitter, you can measure the timer
> interrupt jitter with the "latency" test, but I bet that maybe the
> latency may be higher than 5us;
> - if you really really need to reduce the jitter, then using kernel
> timers instead of sleeping in a user-space thread is recommended,
>
> Yes, you are right that what i have observed is the jitter. I have
> gone through the links which you have provided me in the last
> email. I am building my application using POSIX skin. In the
> links, they have used the rtdm_timer_start(). Now my question is
> that can i use the rtdm functions in the application compiling
> with posix skin, if yes then how. Do i have to change the skin for
> comilation in order to use the rtdm functions. Please refer me
> some guide for using the Xenomai Kernel timers with posix skin in
> order to avoid the jitter. Last thing do i need to made a kernel
> module in order to use the kernel timers. I am confused. Please
> guide me.

If you want to use kernel-space, you need to write a driver, running
in kernel spaces, and drivers use the RTDM API. The POSIX API is for
the application.

--
                                            Gilles.
https://click-hack.org

Re: [Xenomai] Loose determinism when reading pcap and sending as raw ethernet packets

[Gilles Chanteperdrix]

On Thu, Dec 10, 2015 at 05:25:41PM +0000, Umair Ali wrote:
> Hello,
> 
> > >
> > >  what you observe is the jitter, you can measure the timer
> > interrupt jitter with the "latency" test, but I bet that maybe the
> > latency may be higher than 5us;
> > - if you really really need to reduce the jitter, then using kernel
> > timers instead of sleeping in a user-space thread is recommended,
> >
> > Yes, you are right that what i have observed is the jitter. I have
> > gone through the links which you have provided me in the last
> > email. I am building my application using POSIX skin. In the
> > links, they have used the rtdm_timer_start(). Now my question is
> > that can i use the rtdm functions in the application compiling
> > with posix skin, if yes then how. Do i have to change the skin for
> > comilation in order to use the rtdm functions. Please refer me
> > some guide for using the Xenomai Kernel timers with posix skin in
> > order to avoid the jitter. Last thing do i need to made a kernel
> > module in order to use the kernel timers. I am confused. Please
> > guide me.
> 
> > If you want to use kernel-space, you need to write a driver,
> > running in kernel spaces, and drivers use the RTDM API. The POSIX
> > API is for the application.
> 
> As i need to send raw Ethernet packets therefore i am using rtnet
> driver for this purpose. Do you mean that i have to write my own
> driver for ethernet card. Or should i have to write an rtdm driver
> that uses rtnet drivers for sending the raw ethernet. Or is there
> a way i can change the rtnet drivers according to my need.

I mean that if you want to send a packet every 5us with very low
jitter, you should not send the packets from a user-space thread,
but pass them to a driver which sends them based on an RTDM timer. I
would not recommend doing that, because you would have to modify
rtnet to do it: the rtnet stack send path may use a mutex, so can
not be called from a timer handler.

-- 
					    Gilles.
https://click-hack.org