TCP/IP send, sendfile, RAW

TCP/IP send, sendfile, RAW

[Roy Rietveld]

Can somebody help me with this.

I'am new to Linux normaly i do programming for RTOS.

I would like to send ethernet packets with 1400 bytes payload.
I wrote a small program witch sends a buffer of 1400 bytes in a endless 
loop.
The problem is that a would like 100Mbits throughtput but when i check this 
with ethereal.
I only get 40 MBits. I tried sending with an UDP socket and RAW socket. I 
also tried sendfile.
The RAW socket gives the best result till now 50 MBits throughtput.

Is there something faster then send or am i doing something wrong.

I'm running kernel 2.6 on a ARM9 core at 177Mhz 32RAM 32Flash.

Sorry for the bad English.

Thanks

Roy Rietveld

Re: TCP/IP send, sendfile, RAW

[Jan Engelhardt]

> I would like to send ethernet packets with 1400 bytes payload.
> I wrote a small program witch sends a buffer of 1400 bytes in a endless loop.
> The problem is that a would like 100Mbits throughtput but when i check this
> with ethereal.
> I only get 40 MBits. I tried sending with an UDP socket and RAW socket. I also
> tried sendfile.
> The RAW socket gives the best result till now 50 MBits throughtput.

Limitation of Ethernet.



Jan Engelhardt
-- 

Re: TCP/IP send, sendfile, RAW

[linux-os (Dick Johnson)]

On Thu, 4 May 2006, Jan Engelhardt wrote:

>> I would like to send ethernet packets with 1400 bytes payload.
>> I wrote a small program witch sends a buffer of 1400 bytes in a endless loop.
>> The problem is that a would like 100Mbits throughtput but when i check this
>> with ethereal.
>> I only get 40 MBits. I tried sending with an UDP socket and RAW socket. I also
>> tried sendfile.
>> The RAW socket gives the best result till now 50 MBits throughtput.
>
> Limitation of Ethernet.
>
>
>
> Jan Engelhardt

Maybe he can tell what he means by 100 MBits! If he is looking for
100 megabits per second, that's easy, That's 100/8 = 12.5 megabytes
per second. Anything, including Windows on a wet string, will
do that. If he is looking for 100 megabytes per second, that's
hard. He would need 100 * 8 = 800 megabits/second. A "gigabit" link
runs that fast if nobody else is on it, but there is a header and CRC
tail, in addition to the payload. UDP is the protocol to use to realize
this kind of bandwidth, but its possible for some packets to get lost and,
if they are routed, they could even be duplicated. Also, when testing
UDP, there must be a listener in order to realize the high speed.
You can't just spew out a dead-end link.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.4 on an i686 machine (5592.89 BogoMips).
New book: http://www.lymanschool.com
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: TCP/IP send, sendfile, RAW

[Roy Rietveld]

Yes it is 100 MBits and there is a listener. and there are no other pc's on 
the link because its cross cable link. And when sending large buffers 
32Kbyte it will do 80 MBits. It think that there is a lot of overhead in the 
fucntion send or something.


>From: "linux-os (Dick Johnson)" <linux-os@analogic.com>
>Reply-To: "linux-os (Dick Johnson)" <linux-os@analogic.com>
>To: "Jan Engelhardt" <jengelh@linux01.gwdg.de>
>CC: "Roy Rietveld" 
><rwm_rietveld@hotmail.com>,<linux-kernel@vger.kernel.org>
>Subject: Re: TCP/IP send, sendfile, RAW
>Date: Thu, 4 May 2006 13:56:31 -0400
>
>
>On Thu, 4 May 2006, Jan Engelhardt wrote:
>
> >> I would like to send ethernet packets with 1400 bytes payload.
> >> I wrote a small program witch sends a buffer of 1400 bytes in a endless 
>loop.
> >> The problem is that a would like 100Mbits throughtput but when i check 
>this
> >> with ethereal.
> >> I only get 40 MBits. I tried sending with an UDP socket and RAW socket. 
>I also
> >> tried sendfile.
> >> The RAW socket gives the best result till now 50 MBits throughtput.
> >
> > Limitation of Ethernet.
> >
> >
> >
> > Jan Engelhardt
>
>Maybe he can tell what he means by 100 MBits! If he is looking for
>100 megabits per second, that's easy, That's 100/8 = 12.5 megabytes
>per second. Anything, including Windows on a wet string, will
>do that. If he is looking for 100 megabytes per second, that's
>hard. He would need 100 * 8 = 800 megabits/second. A "gigabit" link
>runs that fast if nobody else is on it, but there is a header and CRC
>tail, in addition to the payload. UDP is the protocol to use to realize
>this kind of bandwidth, but its possible for some packets to get lost and,
>if they are routed, they could even be duplicated. Also, when testing
>UDP, there must be a listener in order to realize the high speed.
>You can't just spew out a dead-end link.
>
>Cheers,
>Dick Johnson
>Penguin : Linux version 2.6.16.4 on an i686 machine (5592.89 BogoMips).
>New book: http://www.lymanschool.com
>_
>\x1a\x04
>
>****************************************************************
>The information transmitted in this message is confidential and may be 
>privileged.  Any review, retransmission, dissemination, or other use of 
>this information by persons or entities other than the intended recipient 
>is prohibited.  If you are not the intended recipient, please notify 
>Analogic Corporation immediately - by replying to this message or by 
>sending an email to DeliveryErrors@analogic.com - and destroy all copies of 
>this information, including any attachments, without reading or disclosing 
>them.
>
>Thank you.
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

Re: TCP/IP send, sendfile, RAW

[linux-os (Dick Johnson)]

On Thu, 4 May 2006, Roy Rietveld wrote:

> Yes it is 100 MBits and there is a listener. and there are no other pc's on
> the link because its cross cable link. And when sending large buffers
> 32Kbyte it will do 80 MBits. It think that there is a lot of overhead in the
> fucntion send or something.
>

Use sendto() and recvfrom() for UDP. Stream protocols require an ACK and
are slower.

>
>> From: "linux-os (Dick Johnson)" <linux-os@analogic.com>
>> Reply-To: "linux-os (Dick Johnson)" <linux-os@analogic.com>
>> To: "Jan Engelhardt" <jengelh@linux01.gwdg.de>
>> CC: "Roy Rietveld"
>> <rwm_rietveld@hotmail.com>,<linux-kernel@vger.kernel.org>
>> Subject: Re: TCP/IP send, sendfile, RAW
>> Date: Thu, 4 May 2006 13:56:31 -0400
>>
>>
>> On Thu, 4 May 2006, Jan Engelhardt wrote:
>>
>>>> I would like to send ethernet packets with 1400 bytes payload.
>>>> I wrote a small program witch sends a buffer of 1400 bytes in a endless
>> loop.
>>>> The problem is that a would like 100Mbits throughtput but when i check
>> this
>>>> with ethereal.
>>>> I only get 40 MBits. I tried sending with an UDP socket and RAW socket.
>> I also
>>>> tried sendfile.
>>>> The RAW socket gives the best result till now 50 MBits throughtput.
>>>
>>> Limitation of Ethernet.
>>>
>>>
>>>
>>> Jan Engelhardt
>>
>> Maybe he can tell what he means by 100 MBits! If he is looking for
>> 100 megabits per second, that's easy, That's 100/8 = 12.5 megabytes
>> per second. Anything, including Windows on a wet string, will
>> do that. If he is looking for 100 megabytes per second, that's
>> hard. He would need 100 * 8 = 800 megabits/second. A "gigabit" link
>> runs that fast if nobody else is on it, but there is a header and CRC
>> tail, in addition to the payload. UDP is the protocol to use to realize
>> this kind of bandwidth, but its possible for some packets to get lost and,
>> if they are routed, they could even be duplicated. Also, when testing
>> UDP, there must be a listener in order to realize the high speed.
>> You can't just spew out a dead-end link.
>>
>> Cheers,
>> Dick Johnson
>> Penguin : Linux version 2.6.16.4 on an i686 machine (5592.89 BogoMips).
>> New book: http://www.lymanschool.com
>> _
>> \x1a\x04
>>
>> ****************************************************************
>> The information transmitted in this message is confidential and may be
>> privileged.  Any review, retransmission, dissemination, or other use of
>> this information by persons or entities other than the intended recipient
>> is prohibited.  If you are not the intended recipient, please notify
>> Analogic Corporation immediately - by replying to this message or by
>> sending an email to DeliveryErrors@analogic.com - and destroy all copies of
>> this information, including any attachments, without reading or disclosing
>> them.
>>
>> Thank you.
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>
>
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.4 on an i686 machine (5592.89 BogoMips).
New book: http://www.lymanschool.com
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: TCP/IP send, sendfile, RAW

[Roy Rietveld]

i tried but it doesn't help, still 40MBits. Does send or sento cost a lot of 
cpu load.

i tried to measure the cpu time sendto cost.

gettimeofday(start)
sendto
gettimeofday(end)

print end - start

time measured is 250 us.


>From: "linux-os (Dick Johnson)" <linux-os@analogic.com>
>Reply-To: "linux-os (Dick Johnson)" <linux-os@analogic.com>
>To: "Roy Rietveld" <rwm_rietveld@hotmail.com>
>CC: <linux-kernel@vger.kernel.org>,<jengelh@linux01.gwdg.de>
>Subject: Re: TCP/IP send, sendfile, RAW
>Date: Thu, 4 May 2006 14:27:47 -0400
>
>
>On Thu, 4 May 2006, Roy Rietveld wrote:
>
> > Yes it is 100 MBits and there is a listener. and there are no other pc's 
>on
> > the link because its cross cable link. And when sending large buffers
> > 32Kbyte it will do 80 MBits. It think that there is a lot of overhead in 
>the
> > fucntion send or something.
> >
>
>Use sendto() and recvfrom() for UDP. Stream protocols require an ACK and
>are slower.
>
> >
> >> From: "linux-os (Dick Johnson)" <linux-os@analogic.com>
> >> Reply-To: "linux-os (Dick Johnson)" <linux-os@analogic.com>
> >> To: "Jan Engelhardt" <jengelh@linux01.gwdg.de>
> >> CC: "Roy Rietveld"
> >> <rwm_rietveld@hotmail.com>,<linux-kernel@vger.kernel.org>
> >> Subject: Re: TCP/IP send, sendfile, RAW
> >> Date: Thu, 4 May 2006 13:56:31 -0400
> >>
> >>
> >> On Thu, 4 May 2006, Jan Engelhardt wrote:
> >>
> >>>> I would like to send ethernet packets with 1400 bytes payload.
> >>>> I wrote a small program witch sends a buffer of 1400 bytes in a 
>endless
> >> loop.
> >>>> The problem is that a would like 100Mbits throughtput but when i 
>check
> >> this
> >>>> with ethereal.
> >>>> I only get 40 MBits. I tried sending with an UDP socket and RAW 
>socket.
> >> I also
> >>>> tried sendfile.
> >>>> The RAW socket gives the best result till now 50 MBits throughtput.
> >>>
> >>> Limitation of Ethernet.
> >>>
> >>>
> >>>
> >>> Jan Engelhardt
> >>
> >> Maybe he can tell what he means by 100 MBits! If he is looking for
> >> 100 megabits per second, that's easy, That's 100/8 = 12.5 megabytes
> >> per second. Anything, including Windows on a wet string, will
> >> do that. If he is looking for 100 megabytes per second, that's
> >> hard. He would need 100 * 8 = 800 megabits/second. A "gigabit" link
> >> runs that fast if nobody else is on it, but there is a header and CRC
> >> tail, in addition to the payload. UDP is the protocol to use to realize
> >> this kind of bandwidth, but its possible for some packets to get lost 
>and,
> >> if they are routed, they could even be duplicated. Also, when testing
> >> UDP, there must be a listener in order to realize the high speed.
> >> You can't just spew out a dead-end link.
> >>
> >> Cheers,
> >> Dick Johnson
> >> Penguin : Linux version 2.6.16.4 on an i686 machine (5592.89 BogoMips).
> >> New book: http://www.lymanschool.com
> >> _
> >> \x1a\x04
> >>
> >> ****************************************************************
> >> The information transmitted in this message is confidential and may be
> >> privileged.  Any review, retransmission, dissemination, or other use of
> >> this information by persons or entities other than the intended 
>recipient
> >> is prohibited.  If you are not the intended recipient, please notify
> >> Analogic Corporation immediately - by replying to this message or by
> >> sending an email to DeliveryErrors@analogic.com - and destroy all 
>copies of
> >> this information, including any attachments, without reading or 
>disclosing
> >> them.
> >>
> >> Thank you.
> >> -
> >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" 
>in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> Please read the FAQ at  http://www.tux.org/lkml/
> >
> >
> >
>
>Cheers,
>Dick Johnson
>Penguin : Linux version 2.6.16.4 on an i686 machine (5592.89 BogoMips).
>New book: http://www.lymanschool.com
>_
>\x1a\x04
>
>****************************************************************
>The information transmitted in this message is confidential and may be 
>privileged.  Any review, retransmission, dissemination, or other use of 
>this information by persons or entities other than the intended recipient 
>is prohibited.  If you are not the intended recipient, please notify 
>Analogic Corporation immediately - by replying to this message or by 
>sending an email to DeliveryErrors@analogic.com - and destroy all copies of 
>this information, including any attachments, without reading or disclosing 
>them.
>
>Thank you.

Re: TCP/IP send, sendfile, RAW

[linux-os (Dick Johnson)]

On Thu, 4 May 2006, Roy Rietveld wrote:

> i tried but it doesn't help, still 40MBits. Does send or sento cost a lot of
> cpu load.
>
> i tried to measure the cpu time sendto cost.
>
> gettimeofday(start)
> sendto
> gettimeofday(end)
>
> print end - start
>
> time measured is 250 us.
>

That's probably the granuarity of gettimeofday().

You need to do something like:
         #define USEC 1e6
 	double start_time, end_time, total_time, bytes_per_second;
         get_timeofday(&tb);
         start_time_usec = ((double)tb.tv_sec * USEC) + (double)tb.tv_usec;
 	for(i= 0; i <1000; i++)
             sendto(s, buf, buf_size,...);
         get_timeofday(&tb);
         end_time_usec = ((double)tb.tv_sec * USEC) + (double)tb.tv_usec;
         total_time = (end_time_usec - start_time_usec) / USEC;
         bytes_per_second = (buf_size * 1000) / total_time;


>
>> From: "linux-os (Dick Johnson)" <linux-os@analogic.com>
>> Reply-To: "linux-os (Dick Johnson)" <linux-os@analogic.com>
>> To: "Roy Rietveld" <rwm_rietveld@hotmail.com>
>> CC: <linux-kernel@vger.kernel.org>,<jengelh@linux01.gwdg.de>
>> Subject: Re: TCP/IP send, sendfile, RAW
>> Date: Thu, 4 May 2006 14:27:47 -0400
>>
>>
>> On Thu, 4 May 2006, Roy Rietveld wrote:
>>
>>> Yes it is 100 MBits and there is a listener. and there are no other pc's
>> on
>>> the link because its cross cable link. And when sending large buffers
>>> 32Kbyte it will do 80 MBits. It think that there is a lot of overhead in
>> the
>>> fucntion send or something.
>>>
>>
>> Use sendto() and recvfrom() for UDP. Stream protocols require an ACK and
>> are slower.
>>
>>>
>>>> From: "linux-os (Dick Johnson)" <linux-os@analogic.com>
>>>> Reply-To: "linux-os (Dick Johnson)" <linux-os@analogic.com>
>>>> To: "Jan Engelhardt" <jengelh@linux01.gwdg.de>
>>>> CC: "Roy Rietveld"
>>>> <rwm_rietveld@hotmail.com>,<linux-kernel@vger.kernel.org>
>>>> Subject: Re: TCP/IP send, sendfile, RAW
>>>> Date: Thu, 4 May 2006 13:56:31 -0400
>>>>
>>>>
>>>> On Thu, 4 May 2006, Jan Engelhardt wrote:
>>>>
>>>>>> I would like to send ethernet packets with 1400 bytes payload.
>>>>>> I wrote a small program witch sends a buffer of 1400 bytes in a
>> endless
>>>> loop.
>>>>>> The problem is that a would like 100Mbits throughtput but when i
>> check
>>>> this
>>>>>> with ethereal.
>>>>>> I only get 40 MBits. I tried sending with an UDP socket and RAW
>> socket.
>>>> I also
>>>>>> tried sendfile.
>>>>>> The RAW socket gives the best result till now 50 MBits throughtput.
>>>>>
>>>>> Limitation of Ethernet.
>>>>>
>>>>>
>>>>>
>>>>> Jan Engelhardt
>>>>
>>>> Maybe he can tell what he means by 100 MBits! If he is looking for
>>>> 100 megabits per second, that's easy, That's 100/8 = 12.5 megabytes
>>>> per second. Anything, including Windows on a wet string, will
>>>> do that. If he is looking for 100 megabytes per second, that's
>>>> hard. He would need 100 * 8 = 800 megabits/second. A "gigabit" link
>>>> runs that fast if nobody else is on it, but there is a header and CRC
>>>> tail, in addition to the payload. UDP is the protocol to use to realize
>>>> this kind of bandwidth, but its possible for some packets to get lost
>> and,
>>>> if they are routed, they could even be duplicated. Also, when testing
>>>> UDP, there must be a listener in order to realize the high speed.
>>>> You can't just spew out a dead-end link.
>>>>
>>>> Cheers,
>>>> Dick Johnson
>>>> Penguin : Linux version 2.6.16.4 on an i686 machine (5592.89 BogoMips).
>>>> New book: http://www.lymanschool.com
>>>> _
>>>> \x1a\x04
>>>>
>>>> ****************************************************************
>>>> The information transmitted in this message is confidential and may be
>>>> privileged.  Any review, retransmission, dissemination, or other use of
>>>> this information by persons or entities other than the intended
>> recipient
>>>> is prohibited.  If you are not the intended recipient, please notify
>>>> Analogic Corporation immediately - by replying to this message or by
>>>> sending an email to DeliveryErrors@analogic.com - and destroy all
>> copies of
>>>> this information, including any attachments, without reading or
>> disclosing
>>>> them.
>>>>
>>>> Thank you.
>>>> -
>>>> To unsubscribe from this list: send the line "unsubscribe linux-kernel"
>> in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>> Please read the FAQ at  http://www.tux.org/lkml/
>>>
>>>
>>>
>>
>> Cheers,
>> Dick Johnson
>> Penguin : Linux version 2.6.16.4 on an i686 machine (5592.89 BogoMips).
>> New book: http://www.lymanschool.com
>> _
>> \x1a\x04
>>
>> ****************************************************************
>> The information transmitted in this message is confidential and may be
>> privileged.  Any review, retransmission, dissemination, or other use of
>> this information by persons or entities other than the intended recipient
>> is prohibited.  If you are not the intended recipient, please notify
>> Analogic Corporation immediately - by replying to this message or by
>> sending an email to DeliveryErrors@analogic.com - and destroy all copies of
>> this information, including any attachments, without reading or disclosing
>> them.
>>
>> Thank you.
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.4 on an i686 machine (5592.89 BogoMips).
New book: http://www.lymanschool.com
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: TCP/IP send, sendfile, RAW

[Roy Rietveld]

i Think the resolution of gettimeofday is 1us because
gettimeofday(start)
gettimeofday(end)

end - start gives 1

so when i leave out gettimeofday

while(1)
{
sendto(socket,buffer,1400,....)
}

i measured with ethereal 4000 packets per second
so

1400+headers = 1442

1442*8*4000 = 46MBits

I read something about zero copy may i need somelike that.

>From: "linux-os (Dick Johnson)" <linux-os@analogic.com>
>Reply-To: "linux-os (Dick Johnson)" <linux-os@analogic.com>
>To: "Roy Rietveld" <rwm_rietveld@hotmail.com>
>CC: <linux-kernel@vger.kernel.org>, <jengelh@linux01.gwdg.de>
>Subject: Re: TCP/IP send, sendfile, RAW
>Date: Thu, 4 May 2006 15:12:34 -0400
>
>
>On Thu, 4 May 2006, Roy Rietveld wrote:
>
> > i tried but it doesn't help, still 40MBits. Does send or sento cost a 
>lot of
> > cpu load.
> >
> > i tried to measure the cpu time sendto cost.
> >
> > gettimeofday(start)
> > sendto
> > gettimeofday(end)
> >
> > print end - start
> >
> > time measured is 250 us.
> >
>
>That's probably the granuarity of gettimeofday().
>
>You need to do something like:
>          #define USEC 1e6
>  	double start_time, end_time, total_time, bytes_per_second;
>          get_timeofday(&tb);
>          start_time_usec = ((double)tb.tv_sec * USEC) + 
>(double)tb.tv_usec;
>  	for(i= 0; i <1000; i++)
>              sendto(s, buf, buf_size,...);
>          get_timeofday(&tb);
>          end_time_usec = ((double)tb.tv_sec * USEC) + (double)tb.tv_usec;
>          total_time = (end_time_usec - start_time_usec) / USEC;
>          bytes_per_second = (buf_size * 1000) / total_time;
>
>
> >
> >> From: "linux-os (Dick Johnson)" <linux-os@analogic.com>
> >> Reply-To: "linux-os (Dick Johnson)" <linux-os@analogic.com>
> >> To: "Roy Rietveld" <rwm_rietveld@hotmail.com>
> >> CC: <linux-kernel@vger.kernel.org>,<jengelh@linux01.gwdg.de>
> >> Subject: Re: TCP/IP send, sendfile, RAW
> >> Date: Thu, 4 May 2006 14:27:47 -0400
> >>
> >>
> >> On Thu, 4 May 2006, Roy Rietveld wrote:
> >>
> >>> Yes it is 100 MBits and there is a listener. and there are no other 
>pc's
> >> on
> >>> the link because its cross cable link. And when sending large buffers
> >>> 32Kbyte it will do 80 MBits. It think that there is a lot of overhead 
>in
> >> the
> >>> fucntion send or something.
> >>>
> >>
> >> Use sendto() and recvfrom() for UDP. Stream protocols require an ACK 
>and
> >> are slower.
> >>
> >>>
> >>>> From: "linux-os (Dick Johnson)" <linux-os@analogic.com>
> >>>> Reply-To: "linux-os (Dick Johnson)" <linux-os@analogic.com>
> >>>> To: "Jan Engelhardt" <jengelh@linux01.gwdg.de>
> >>>> CC: "Roy Rietveld"
> >>>> <rwm_rietveld@hotmail.com>,<linux-kernel@vger.kernel.org>
> >>>> Subject: Re: TCP/IP send, sendfile, RAW
> >>>> Date: Thu, 4 May 2006 13:56:31 -0400
> >>>>
> >>>>
> >>>> On Thu, 4 May 2006, Jan Engelhardt wrote:
> >>>>
> >>>>>> I would like to send ethernet packets with 1400 bytes payload.
> >>>>>> I wrote a small program witch sends a buffer of 1400 bytes in a
> >> endless
> >>>> loop.
> >>>>>> The problem is that a would like 100Mbits throughtput but when i
> >> check
> >>>> this
> >>>>>> with ethereal.
> >>>>>> I only get 40 MBits. I tried sending with an UDP socket and RAW
> >> socket.
> >>>> I also
> >>>>>> tried sendfile.
> >>>>>> The RAW socket gives the best result till now 50 MBits throughtput.
> >>>>>
> >>>>> Limitation of Ethernet.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Jan Engelhardt
> >>>>
> >>>> Maybe he can tell what he means by 100 MBits! If he is looking for
> >>>> 100 megabits per second, that's easy, That's 100/8 = 12.5 megabytes
> >>>> per second. Anything, including Windows on a wet string, will
> >>>> do that. If he is looking for 100 megabytes per second, that's
> >>>> hard. He would need 100 * 8 = 800 megabits/second. A "gigabit" link
> >>>> runs that fast if nobody else is on it, but there is a header and CRC
> >>>> tail, in addition to the payload. UDP is the protocol to use to 
>realize
> >>>> this kind of bandwidth, but its possible for some packets to get lost
> >> and,
> >>>> if they are routed, they could even be duplicated. Also, when testing
> >>>> UDP, there must be a listener in order to realize the high speed.
> >>>> You can't just spew out a dead-end link.
> >>>>
> >>>> Cheers,
> >>>> Dick Johnson
> >>>> Penguin : Linux version 2.6.16.4 on an i686 machine (5592.89 
>BogoMips).
> >>>> New book: http://www.lymanschool.com
> >>>> _
> >>>> \x1a\x04
> >>>>
> >>>> ****************************************************************
> >>>> The information transmitted in this message is confidential and may 
>be
> >>>> privileged.  Any review, retransmission, dissemination, or other use 
>of
> >>>> this information by persons or entities other than the intended
> >> recipient
> >>>> is prohibited.  If you are not the intended recipient, please notify
> >>>> Analogic Corporation immediately - by replying to this message or by
> >>>> sending an email to DeliveryErrors@analogic.com - and destroy all
> >> copies of
> >>>> this information, including any attachments, without reading or
> >> disclosing
> >>>> them.
> >>>>
> >>>> Thank you.
> >>>> -
> >>>> To unsubscribe from this list: send the line "unsubscribe 
>linux-kernel"
> >> in
> >>>> the body of a message to majordomo@vger.kernel.org
> >>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>> Please read the FAQ at  http://www.tux.org/lkml/
> >>>
> >>>
> >>>
> >>
> >> Cheers,
> >> Dick Johnson
> >> Penguin : Linux version 2.6.16.4 on an i686 machine (5592.89 BogoMips).
> >> New book: http://www.lymanschool.com
> >> _
> >> \x1a\x04
> >>
> >> ****************************************************************
> >> The information transmitted in this message is confidential and may be
> >> privileged.  Any review, retransmission, dissemination, or other use of
> >> this information by persons or entities other than the intended 
>recipient
> >> is prohibited.  If you are not the intended recipient, please notify
> >> Analogic Corporation immediately - by replying to this message or by
> >> sending an email to DeliveryErrors@analogic.com - and destroy all 
>copies of
> >> this information, including any attachments, without reading or 
>disclosing
> >> them.
> >>
> >> Thank you.
> >
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" 
>in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> >
>
>Cheers,
>Dick Johnson
>Penguin : Linux version 2.6.16.4 on an i686 machine (5592.89 BogoMips).
>New book: http://www.lymanschool.com
>_
>\x1a\x04
>
>****************************************************************
>The information transmitted in this message is confidential and may be 
>privileged.  Any review, retransmission, dissemination, or other use of 
>this information by persons or entities other than the intended recipient 
>is prohibited.  If you are not the intended recipient, please notify 
>Analogic Corporation immediately - by replying to this message or by 
>sending an email to DeliveryErrors@analogic.com - and destroy all copies of 
>this information, including any attachments, without reading or disclosing 
>them.
>
>Thank you.
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

Re: TCP/IP send, sendfile, RAW

[Willy Tarreau]

On Thu, May 04, 2006 at 07:43:34PM +0000, Roy Rietveld wrote:
> i Think the resolution of gettimeofday is 1us because
> gettimeofday(start)
> gettimeofday(end)
> 
> end - start gives 1
> 
> so when i leave out gettimeofday
> 
> while(1)
> {
> sendto(socket,buffer,1400,....)
> }
> 
> i measured with ethereal 4000 packets per second

Are you sure you are not limited by the interrupt rate ? may be interrupt
processing causes a large overhead at 4000/s for 177 MHz, and you might
need to find a way to either use interrupt mitigation or polling.

Regards,
Willy

Re: TCP/IP send, sendfile, RAW

[Phillip Susi]

Roy Rietveld wrote:
> I read something about zero copy may i need somelike that.

Yes, zero copy IO does make a huge difference.  Unfortunately, at this 
time, it is not possible to do zero copy IO on sockets save for 
sendfile() because they do not yet support aio, at least, not the last 
time I checked.

For comparison I wrote an ftp server for NT 4.0 several years ago on a 
PII-233 system ( similar speed to yours ) and saw similar results to 
yours until switching to zero copy IO, which pushed 11,820 KB/s ftp 
transfers using less than 1% of the cpu.

I'm still waiting for Linux to be able to do this, and hopefully won't 
have to wait much longer.

Re: TCP/IP send, sendfile, RAW

[Alan Cox]

On Iau, 2006-05-04 at 18:42 +0000, Roy Rietveld wrote:
> i tried but it doesn't help, still 40MBits. Does send or sento cost a lot of 
> cpu load.

That depends a lot on your hardware: Each UDP or TCP packet has to be
checksummed  and loaded onto the network chip so there is memory
bandwidth used. There is also some CPU load but they are not horribly
complex code paths. Much of the performance depend son the network chip

Re: TCP/IP send, sendfile, RAW

[Denis Vlasenko]

On Thursday 04 May 2006 20:19, Roy Rietveld wrote:
> Can somebody help me with this.
> 
> I'am new to Linux normaly i do programming for RTOS.
> 
> I would like to send ethernet packets with 1400 bytes payload.
> I wrote a small program witch sends a buffer of 1400 bytes in a endless 
> loop.
> The problem is that a would like 100Mbits throughtput but when i check this 
> with ethereal.
> I only get 40 MBits. I tried sending with an UDP socket and RAW socket. I 
> also tried sendfile.
> The RAW socket gives the best result till now 50 MBits throughtput.
> 
> Is there something faster then send or am i doing something wrong.

Get netcat source and read it. It's small.

netcat is definitely able to saturate 100Mbit link with both TCP and UDP.
--
vda

Re: TCP/IP send, sendfile, RAW

[Roy Rietveld]

the platform is a netsilicon NS9360 witch include an 100MBit ethernet 
device.
The driver came with the software(LxNETES) for the development board.
CPU load is 100% when running sender program.

cat /proc/interrupts; sleep 10; cat /proc/interrupts doen't work anymore 
because cpu is to busy.

Does sendto give other processes time when the hardware is transmitting 
data?
Is this bad hardware or is the cost of sendto that high.



>From: Pádraig Brady <P@draigBrady.com>
>To: Roy Rietveld <rwm_rietveld@hotmail.com>
>Subject: Re: TCP/IP send, sendfile, RAW
>Date: Fri, 05 May 2006 10:09:49 +0100
>
>Roy Rietveld wrote:
>
> > Can somebody help me with this.
> >
> > I'am new to Linux normaly i do programming for RTOS.
> >
> > I would like to send ethernet packets with 1400 bytes payload.
> > I wrote a small program witch sends a buffer of 1400 bytes in a
> > endless loop.
> > The problem is that a would like 100Mbits throughtput but when i check
> > this with ethereal.
> > I only get 40 MBits. I tried sending with an UDP socket and RAW
> > socket. I also tried sendfile.
> > The RAW socket gives the best result till now 50 MBits throughtput.
> >
> > Is there something faster then send or am i doing something wrong.
> >
> > I'm running kernel 2.6 on a ARM9 core at 177Mhz 32RAM 32Flash.
>
>Is this the platform for both sender and receiver?
>
>What you have to consider is processing required per packet.
>At 50Mb/s you are getting for following number of packets per second:
>
>$ echo "(50*10^6)/8/(12+8+14+20+8+1400+4)" | bc
>4263
>
>I'm guessing that the receiver or sender is running out of CPU at this 
>rate?
>Also maybe this number of interrupts/s may be an issue on this platform?
>You can check the interrupt rate easilty enough with:
>cat /proc/interrupts; sleep 10; cat /proc/interrupts
>
>Note by default ethereal (libpcap) will use 3 syscalls per packet
>to copy and timestamp each packet. Have a look at PACKET_MMAP
>to alleviate this.
>
>At the interrupt level you could use NAPI or interrupt coalescing etc.
>What driver are you using?
>
>Pádraig.
>

Re: TCP/IP send, sendfile, RAW

[Avi Kivity]

Roy Rietveld wrote:
> the platform is a netsilicon NS9360 witch include an 100MBit ethernet 
> device.
> The driver came with the software(LxNETES) for the development board.
> CPU load is 100% when running sender program.
>
> cat /proc/interrupts; sleep 10; cat /proc/interrupts doen't work 
> anymore because cpu is to busy.
>
> Does sendto give other processes time when the hardware is 
> transmitting data?
> Is this bad hardware or is the cost of sendto that high.

Try sending UDP over loopback and see.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: TCP/IP send, sendfile, RAW

[Roy Rietveld]

Thanks now i know, the stack is slow 40MBit on loopback.


>From: Avi Kivity <avi@argo.co.il>
>To: Roy Rietveld <rwm_rietveld@hotmail.com>
>CC: P@draigBrady.com, linux-kernel@vger.kernel.org
>Subject: Re: TCP/IP send, sendfile, RAW
>Date: Fri, 05 May 2006 15:39:56 +0300
>
>Roy Rietveld wrote:
>>the platform is a netsilicon NS9360 witch include an 100MBit ethernet 
>>device.
>>The driver came with the software(LxNETES) for the development board.
>>CPU load is 100% when running sender program.
>>
>>cat /proc/interrupts; sleep 10; cat /proc/interrupts doen't work anymore 
>>because cpu is to busy.
>>
>>Does sendto give other processes time when the hardware is transmitting 
>>data?
>>Is this bad hardware or is the cost of sendto that high.
>
>Try sending UDP over loopback and see.
>
>--
>Do not meddle in the internals of kernels, for they are subtle and quick to 
>panic.
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

Re: TCP/IP send, sendfile, RAW

[Mark Lord]

Roy Rietveld wrote:
> Yes it is 100 MBits and there is a listener. and there are no other pc's 
> on the link because its cross cable link. And when sending large buffers 
> 32Kbyte it will do 80 MBits. It think that there is a lot of overhead in 
> the fucntion send or something.

I'm not sure what the problem is here.  I just now cobbled together 
a pair of programs using SOCK_RAW, and ran the sender on my notebook
and the receiver on a 600Mhz VIA EPIA box, with a consumer grade 100mb/sec
switch in the middle.

Throughput was 98.4 mbits/sec using 1400 byte buffers.

Cheers