Hight speed data sending from custom IP out of kernel

Hight speed data sending from custom IP out of kernel

[Michal Simek]

Hi,

I would like to create demo for high speed data sending from custom IP through 
the ethernet. I think the best description is that there are dmaable memory 
mapped registers or just memory which store data I want to send (for example 200MB).
Linux should handle all communication between target(probably server) and host 
(client) but data in the packets should go from that custom IP and can't go 
through the kernel because of performance issue.

Ethernet core have own DMA which I could use but the question is if there is any 
option how to convince the kernel that data will go directly from memory mapped 
registers and the kernel/driver/... just setup dma BD for headers and second for 
data.

Do you have any experience with any solution with passing data completely out of 
kernel?

Thanks,
Michal

-- 
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian

Re: Hight speed data sending from custom IP out of kernel

[juice]

Hi Michal.

How fast do you need to send the data?
I have an application where I send test stream out to GE line and can fill
the total capacity of the ethernet regardless of the packet size.

The test stream I am sending is stored in kernel memory, and therefore is
limited by the amount of free memory. 200M is no problem.

The solution I am using is loosely based on the pktgen module, except that
my module can load a wireshark capture from userland program and then send
it from ethernet interface in wire speed.

  - Juice -


> Hi,
> I would like to create demo for high speed data sending from custom IP
through
> the ethernet. I think the best description is that there are dmaable memory
> mapped registers or just memory which store data I want to send (for
example 200MB).
> Linux should handle all communication between target(probably server)
and
> host
> (client) but data in the packets should go from that custom IP and can't go
> through the kernel because of performance issue.
> Ethernet core have own DMA which I could use but the question is if
there
> is any
> option how to convince the kernel that data will go directly from memory
mapped
> registers and the kernel/driver/... just setup dma BD for headers and
second for
> data.
> Do you have any experience with any solution with passing data
completely
> out of
> kernel?
> Thanks,
> Michal
> --
> Michal Simek, Ing. (M.Eng)
> w: www.monstr.eu p: +42-0-721842854
> Maintainer of Linux kernel 2.6 Microblaze Linux -
> http://www.monstr.eu/fdt/
> Microblaze U-BOOT custodian
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in the
body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Hight speed data sending from custom IP out of kernel

[Michal Simek]

Hi Juice,

juice wrote:
> Hi Michal.
> 
> How fast do you need to send the data?

It sounds weird but as fast as possible. There is no specific limit because I 
want to create demo and test it on various hw configuration which I can easily 
create on FPGA. For now the bottleneck is Microblaze cpu. It can run from 50MHz 
till 170-180MHz. We also support both endians and have two hw IP 
cores(10/100/1000) which I can use.

> I have an application where I send test stream out to GE line and can fill
> the total capacity of the ethernet regardless of the packet size.

What cpu do you use?

> 
> The test stream I am sending is stored in kernel memory, and therefore is
> limited by the amount of free memory. 200M is no problem.

Is it UDP or TCP?

> 
> The solution I am using is loosely based on the pktgen module, except that
> my module can load a wireshark capture from userland program and then send
> it from ethernet interface in wire speed.

Sound good. Would it be possible to see it and test it?

Thanks,
Michal


> 
>   - Juice -
> 
> 
>> Hi,
>> I would like to create demo for high speed data sending from custom IP
> through
>> the ethernet. I think the best description is that there are dmaable memory
>> mapped registers or just memory which store data I want to send (for
> example 200MB).
>> Linux should handle all communication between target(probably server)
> and
>> host
>> (client) but data in the packets should go from that custom IP and can't go
>> through the kernel because of performance issue.
>> Ethernet core have own DMA which I could use but the question is if
> there
>> is any
>> option how to convince the kernel that data will go directly from memory
> mapped
>> registers and the kernel/driver/... just setup dma BD for headers and
> second for
>> data.
>> Do you have any experience with any solution with passing data
> completely
>> out of
>> kernel?
>> Thanks,
>> Michal
>> --
>> Michal Simek, Ing. (M.Eng)
>> w: www.monstr.eu p: +42-0-721842854
>> Maintainer of Linux kernel 2.6 Microblaze Linux -
>> http://www.monstr.eu/fdt/
>> Microblaze U-BOOT custodian
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in the
> body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 
> 


-- 
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian

Re: Hight speed data sending from custom IP out of kernel

[juice]

Hi!

I can see you are probably going to run into CPU performance problems, but
it depends a lot on the type of traffic you are going to send.

My system requires quite fast processor, but even more important is to
have a network interface card that really supports the full speed of
gigabit ethernet line. The reason for that is that my test traffic
includes streams of very small packets that cause a lot of overhead in
processing.

Most of my test traffic is UDP, but it does not really matter what the
higher layers of the traffic are, this scheme operates on the ethernet
layer and does not care about payload structure.

I tried several NIC:s before i settled using Intel 82576 cards with the
igb driver. If you have less capable interface card, your small packet
performance is going to be a lot poorer.

Using that card I can get to full speed GE line rate even with 64byte
packets, but if you want to send larger packets, say close to 1500byte
then almost any NIC will work OK for you.

You can download the module code and the userland seeding application from
my svn server at https://toosa.swagman.org/svn/streamgen
The streamseed userland application requires libpcap-dev to build
correctly but the streamgen module is self-sufficent.

There is not a lot of documentation, and the module is still "work in
progress" as I am going to fix it to work with more than one interface at
the same time when I get to do it. Currently it can only use one interface
on the sending host machine.

  - Juice -


> Hi Juice,
>
> juice wrote:
>> Hi Michal.
>>
>> How fast do you need to send the data?
>
> It sounds weird but as fast as possible. There is no specific limit
> because I
> want to create demo and test it on various hw configuration which I can
> easily
> create on FPGA. For now the bottleneck is Microblaze cpu. It can run from
> 50MHz
> till 170-180MHz. We also support both endians and have two hw IP
> cores(10/100/1000) which I can use.
>
>> I have an application where I send test stream out to GE line and can
>> fill
>> the total capacity of the ethernet regardless of the packet size.
>
> What cpu do you use?
>
>>
>> The test stream I am sending is stored in kernel memory, and therefore
>> is
>> limited by the amount of free memory. 200M is no problem.
>
> Is it UDP or TCP?
>
>>
>> The solution I am using is loosely based on the pktgen module, except
>> that
>> my module can load a wireshark capture from userland program and then
>> send
>> it from ethernet interface in wire speed.
>
> Sound good. Would it be possible to see it and test it?
>
> Thanks,
> Michal
>
>
>>
>>   - Juice -
>>
>>
>>> Hi,
>>> I would like to create demo for high speed data sending from custom IP
>> through
>>> the ethernet. I think the best description is that there are dmaable
>>> memory
>>> mapped registers or just memory which store data I want to send (for
>> example 200MB).
>>> Linux should handle all communication between target(probably server)
>> and
>>> host
>>> (client) but data in the packets should go from that custom IP and
>>> can't go
>>> through the kernel because of performance issue.
>>> Ethernet core have own DMA which I could use but the question is if
>> there
>>> is any
>>> option how to convince the kernel that data will go directly from
>>> memory
>> mapped
>>> registers and the kernel/driver/... just setup dma BD for headers and
>> second for
>>> data.
>>> Do you have any experience with any solution with passing data
>> completely
>>> out of
>>> kernel?
>>> Thanks,
>>> Michal
>>> --
>>> Michal Simek, Ing. (M.Eng)
>>> w: www.monstr.eu p: +42-0-721842854
>>> Maintainer of Linux kernel 2.6 Microblaze Linux -
>>> http://www.monstr.eu/fdt/
>>> Microblaze U-BOOT custodian
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>> the
>> body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>>
>
>
> --
> Michal Simek, Ing. (M.Eng)
> w: www.monstr.eu p: +42-0-721842854
> Maintainer of Linux kernel 2.6 Microblaze Linux -
> http://www.monstr.eu/fdt/
> Microblaze U-BOOT custodian
>

Re: Hight speed data sending from custom IP out of kernel

[Michal Simek]

juice wrote:
> Hi!
> 
> I can see you are probably going to run into CPU performance problems, but
> it depends a lot on the type of traffic you are going to send.
> 
> My system requires quite fast processor, but even more important is to
> have a network interface card that really supports the full speed of
> gigabit ethernet line. The reason for that is that my test traffic
> includes streams of very small packets that cause a lot of overhead in
> processing.
> 
> Most of my test traffic is UDP, but it does not really matter what the
> higher layers of the traffic are, this scheme operates on the ethernet
> layer and does not care about payload structure.
> 
> I tried several NIC:s before i settled using Intel 82576 cards with the
> igb driver. If you have less capable interface card, your small packet
> performance is going to be a lot poorer.
> 
> Using that card I can get to full speed GE line rate even with 64byte
> packets, but if you want to send larger packets, say close to 1500byte
> then almost any NIC will work OK for you.
> 
> You can download the module code and the userland seeding application from
> my svn server at https://toosa.swagman.org/svn/streamgen
> The streamseed userland application requires libpcap-dev to build
> correctly but the streamgen module is self-sufficent.
> 
> There is not a lot of documentation, and the module is still "work in
> progress" as I am going to fix it to work with more than one interface at
> the same time when I get to do it. Currently it can only use one interface
> on the sending host machine.

Thanks for that. I am looking at pktgen. On UDP my system is able to send full 
bandwidth on 100Mbit/s ethernet and 220Mbit/s on 1G/s.
I will let you know when I have any useful resutls.

Thanks,
Michal

> 
>   - Juice -
> 
> 
>> Hi Juice,
>>
>> juice wrote:
>>> Hi Michal.
>>>
>>> How fast do you need to send the data?
>> It sounds weird but as fast as possible. There is no specific limit
>> because I
>> want to create demo and test it on various hw configuration which I can
>> easily
>> create on FPGA. For now the bottleneck is Microblaze cpu. It can run from
>> 50MHz
>> till 170-180MHz. We also support both endians and have two hw IP
>> cores(10/100/1000) which I can use.
>>
>>> I have an application where I send test stream out to GE line and can
>>> fill
>>> the total capacity of the ethernet regardless of the packet size.
>> What cpu do you use?
>>
>>> The test stream I am sending is stored in kernel memory, and therefore
>>> is
>>> limited by the amount of free memory. 200M is no problem.
>> Is it UDP or TCP?
>>
>>> The solution I am using is loosely based on the pktgen module, except
>>> that
>>> my module can load a wireshark capture from userland program and then
>>> send
>>> it from ethernet interface in wire speed.
>> Sound good. Would it be possible to see it and test it?
>>
>> Thanks,
>> Michal
>>
>>
>>>   - Juice -
>>>
>>>
>>>> Hi,
>>>> I would like to create demo for high speed data sending from custom IP
>>> through
>>>> the ethernet. I think the best description is that there are dmaable
>>>> memory
>>>> mapped registers or just memory which store data I want to send (for
>>> example 200MB).
>>>> Linux should handle all communication between target(probably server)
>>> and
>>>> host
>>>> (client) but data in the packets should go from that custom IP and
>>>> can't go
>>>> through the kernel because of performance issue.
>>>> Ethernet core have own DMA which I could use but the question is if
>>> there
>>>> is any
>>>> option how to convince the kernel that data will go directly from
>>>> memory
>>> mapped
>>>> registers and the kernel/driver/... just setup dma BD for headers and
>>> second for
>>>> data.
>>>> Do you have any experience with any solution with passing data
>>> completely
>>>> out of
>>>> kernel?
>>>> Thanks,
>>>> Michal
>>>> --
>>>> Michal Simek, Ing. (M.Eng)
>>>> w: www.monstr.eu p: +42-0-721842854
>>>> Maintainer of Linux kernel 2.6 Microblaze Linux -
>>>> http://www.monstr.eu/fdt/
>>>> Microblaze U-BOOT custodian
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>>> the
>>> body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>>
>>
>> --
>> Michal Simek, Ing. (M.Eng)
>> w: www.monstr.eu p: +42-0-721842854
>> Maintainer of Linux kernel 2.6 Microblaze Linux -
>> http://www.monstr.eu/fdt/
>> Microblaze U-BOOT custodian
>>
> 
> 


-- 
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian

Re: Hight speed data sending from custom IP out of kernel

[Eric Dumazet]

Le jeudi 21 avril 2011 à 10:02 +0200, Michal Simek a écrit :

> Thanks for that. I am looking at pktgen. On UDP my system is able to send full 
> bandwidth on 100Mbit/s ethernet and 220Mbit/s on 1G/s.
> I will let you know when I have any useful resutls.

220Mbits/s in pktgen or an application ?
- how many packets per second ? (or packet size ?)

pktgen has the "clone_skb 100" thing that avoid skb_alloc()/skb_free()
overhead, and permits to really test driver performance.

It also bypass qdisc management.

Re: Hight speed data sending from custom IP out of kernel

[Michal Simek]

Eric Dumazet wrote:
> Le jeudi 21 avril 2011 à 10:02 +0200, Michal Simek a écrit :
> 
>> Thanks for that. I am looking at pktgen. On UDP my system is able to send full 
>> bandwidth on 100Mbit/s ethernet and 220Mbit/s on 1G/s.
>> I will let you know when I have any useful resutls.
> 
> 220Mbits/s in pktgen or an application ?
> - how many packets per second ? (or packet size ?)

pktgen - Packet size 1500 on 1Gb/s lan - 66MHx Microblaze cpu without hwcsum 
support.

~ # cat /proc/net/pktgen/eth0
Params: count 10000000  min_pkt_size: 1500  max_pkt_size: 1500
      frags: 0  delay: 0  clone_skb: 1000000  ifname: eth0
      flows: 0 flowlen: 0
      queue_map_min: 0  queue_map_max: 0
      dst_min: 192.168.0.102  dst_max:
         src_min:   src_max:
      src_mac: 00:0a:35:00:8d:0d dst_mac: 00:12:79:c0:59:15
      udp_src_min: 9  udp_src_max: 9  udp_dst_min: 9  udp_dst_max: 9
      src_mac_count: 0  dst_mac_count: 0
      Flags:
Current:
      pkts-sofar: 254875  errors: 0
      started: 729900351us  stopped: 743064495us idle: 26885us
      seq_num: 254876  cur_dst_mac_offset: 0  cur_src_mac_offset: 0
      cur_saddr: 0xc0a80073  cur_daddr: 0xc0a80066
      cur_udp_dst: 9  cur_udp_src: 9
      cur_queue_map: 0
      flows: 0
Result: OK: 13164144(c13137258+d26885) nsec, 254875 (1500byte,0frags)
   19361pps 232Mb/sec (232332000bps) errors: 0

> 
> pktgen has the "clone_skb 100" thing that avoid skb_alloc()/skb_free()
> overhead, and permits to really test driver performance.
> 
> It also bypass qdisc management.

I have also tried to removed fragments support from the driver to find out what 
happen if I remove the part of code from xmit path and there is improvement.

~ # cat /proc/net/pktgen/eth0
Params: count 10000000  min_pkt_size: 1500  max_pkt_size: 1500
      frags: 0  delay: 0  clone_skb: 1000000  ifname: eth0
      flows: 0 flowlen: 0
      queue_map_min: 0  queue_map_max: 0
      dst_min: 192.168.0.102  dst_max:
         src_min:   src_max:
      src_mac: 00:0a:35:00:8d:0d dst_mac: 00:12:79:c0:59:15
      udp_src_min: 9  udp_src_max: 9  udp_dst_min: 9  udp_dst_max: 9
      src_mac_count: 0  dst_mac_count: 0
      Flags:
Current:
      pkts-sofar: 2041908  errors: 0
      started: 45216342us  stopped: 146752402us idle: 135633us
      seq_num: 2041909  cur_dst_mac_offset: 0  cur_src_mac_offset: 0
      cur_saddr: 0xc0a80073  cur_daddr: 0xc0a80066
      cur_udp_dst: 9  cur_udp_src: 9
      cur_queue_map: 0
      flows: 0
Result: OK: 101536059(c101400425+d135633) nsec, 2041908 (1500byte,0frags)
   20110pps 241Mb/sec (241320000bps) errors: 0


I have also tested frags support and here are results for above configuration.
frags 0 = 232Mb/s
frags 1 = 192Mb/s
frags 2 = 159Mb/s
frags 3 = 141Mb/s
frags 4 = 130Mb/s
frags 5 = 116Mb/s

The eth controller generate irq when tx is done. I measured it yesterday and it 
takes 810 timer ticks to send data (1.5k packet size). It is hard to expect how 
many cpu instructions it is but maybe half that's why I want to try to disable 
IRQ generation and wait when DMA copy is done. Do I have any other options?

Thanks,
Michal

-- 
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian

Re: Hight speed data sending from custom IP out of kernel

[Michal Simek]

Hi,

Eric Dumazet wrote:
> Le jeudi 21 avril 2011 à 10:02 +0200, Michal Simek a écrit :
> 
>> Thanks for that. I am looking at pktgen. On UDP my system is able to send full 
>> bandwidth on 100Mbit/s ethernet and 220Mbit/s on 1G/s.
>> I will let you know when I have any useful resutls.
> 
> 220Mbits/s in pktgen or an application ?
> - how many packets per second ? (or packet size ?)
> 
> pktgen has the "clone_skb 100" thing that avoid skb_alloc()/skb_free()
> overhead, and permits to really test driver performance.
> 
> It also bypass qdisc management.
> 

I have reused the part of code from pktgen and I have found that I am missing 
some IDs that's why I have done one simple patch(below) in pktgen which is 
update IP ID field to find out if all packets are sent or not. As you suggest I 
am also missing some IDs here.
My question is if I can use any mechanism to ensure to sending all IDs?

The next my question about packet fragments. Is it possible to setup IP 
fragments from higher level? I do it on low level as pktgen and I have change 
page address to memory which I need to send but it in under UDP.

The point is to create packet with frags > 1 where the first fragment is IP/TCP 
header and the second fragments contains pointer to data which are prepared in 
the memory and will be copied directly by network driver. I am doing the same 
hacked code from pktgen. Is it possible to do it on higher level?

Thanks,
Michal


For 2.6.37.6
diff --git a/net/core/pktgen.c b/net/core/pktgen.c
index 33bc382..3429eb3 100644
--- a/net/core/pktgen.c
+++ b/net/core/pktgen.c
@@ -3500,6 +3500,13 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
                 pkt_dev->last_pkt_size = pkt_dev->skb->len;
                 pkt_dev->allocated_skbs++;
                 pkt_dev->clone_count = 0;       /* reset counter */
+       } else {
+               struct iphdr *iph;
+               iph = ip_hdr(pkt_dev->skb);
+               iph->id = htons(pkt_dev->ip_id);
+               pkt_dev->ip_id++;
+               iph->check = 0;
+               iph->check = ip_fast_csum((void *)iph, iph->ihl);
         }

         if (pkt_dev->delay && pkt_dev->last_ok)



-- 
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian

Re: Hight speed data sending from custom IP out of kernel

[Eric Dumazet]

Le lundi 25 avril 2011 à 13:18 +0200, Michal Simek a écrit :
> Hi,
> 
> Eric Dumazet wrote:
> > Le jeudi 21 avril 2011 à 10:02 +0200, Michal Simek a écrit :
> > 
> >> Thanks for that. I am looking at pktgen. On UDP my system is able to send full 
> >> bandwidth on 100Mbit/s ethernet and 220Mbit/s on 1G/s.
> >> I will let you know when I have any useful resutls.
> > 
> > 220Mbits/s in pktgen or an application ?
> > - how many packets per second ? (or packet size ?)
> > 
> > pktgen has the "clone_skb 100" thing that avoid skb_alloc()/skb_free()
> > overhead, and permits to really test driver performance.
> > 
> > It also bypass qdisc management.
> > 
> 
> I have reused the part of code from pktgen and I have found that I am missing 
> some IDs that's why I have done one simple patch(below) in pktgen which is 
> update IP ID field to find out if all packets are sent or not. As you suggest I 
> am also missing some IDs here.
> My question is if I can use any mechanism to ensure to sending all IDs?
> 
> The next my question about packet fragments. Is it possible to setup IP 
> fragments from higher level? I do it on low level as pktgen and I have change 
> page address to memory which I need to send but it in under UDP.
> 
> The point is to create packet with frags > 1 where the first fragment is IP/TCP 
> header and the second fragments contains pointer to data which are prepared in 
> the memory and will be copied directly by network driver. I am doing the same 
> hacked code from pktgen. Is it possible to do it on higher level?
> 

sendfile() is mostly doing this.

> Thanks,
> Michal
> 
> 
> For 2.6.37.6
> diff --git a/net/core/pktgen.c b/net/core/pktgen.c
> index 33bc382..3429eb3 100644
> --- a/net/core/pktgen.c
> +++ b/net/core/pktgen.c
> @@ -3500,6 +3500,13 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
>                  pkt_dev->last_pkt_size = pkt_dev->skb->len;
>                  pkt_dev->allocated_skbs++;
>                  pkt_dev->clone_count = 0;       /* reset counter */
> +       } else {
> +               struct iphdr *iph;
> +               iph = ip_hdr(pkt_dev->skb);
> +               iph->id = htons(pkt_dev->ip_id);
> +               pkt_dev->ip_id++;
> +               iph->check = 0;
> +               iph->check = ip_fast_csum((void *)iph, iph->ihl);
>          }
> 
>          if (pkt_dev->delay && pkt_dev->last_ok)
> 
> 
> 


Well, you cant do that in pktgen, since you're changing previous packet
content (it might still be in device TX queue, not yet sent, or being
sent right now)

Now, if all you want to do is send many packets from pktgen (with only
ID changing), you could add a fast path to not rebuild from scratch new
packets.

Re: Hight speed data sending from custom IP out of kernel

[Michal Simek]

Eric Dumazet wrote:
> Le lundi 25 avril 2011 à 13:18 +0200, Michal Simek a écrit :
>> Hi,
>>
>> Eric Dumazet wrote:
>>> Le jeudi 21 avril 2011 à 10:02 +0200, Michal Simek a écrit :
>>>
>>>> Thanks for that. I am looking at pktgen. On UDP my system is able to send full 
>>>> bandwidth on 100Mbit/s ethernet and 220Mbit/s on 1G/s.
>>>> I will let you know when I have any useful resutls.
>>> 220Mbits/s in pktgen or an application ?
>>> - how many packets per second ? (or packet size ?)
>>>
>>> pktgen has the "clone_skb 100" thing that avoid skb_alloc()/skb_free()
>>> overhead, and permits to really test driver performance.
>>>
>>> It also bypass qdisc management.
>>>
>> I have reused the part of code from pktgen and I have found that I am missing 
>> some IDs that's why I have done one simple patch(below) in pktgen which is 
>> update IP ID field to find out if all packets are sent or not. As you suggest I 
>> am also missing some IDs here.
>> My question is if I can use any mechanism to ensure to sending all IDs?
>>
>> The next my question about packet fragments. Is it possible to setup IP 
>> fragments from higher level? I do it on low level as pktgen and I have change 
>> page address to memory which I need to send but it in under UDP.
>>
>> The point is to create packet with frags > 1 where the first fragment is IP/TCP 
>> header and the second fragments contains pointer to data which are prepared in 
>> the memory and will be copied directly by network driver. I am doing the same 
>> hacked code from pktgen. Is it possible to do it on higher level?
>>
> 
> sendfile() is mostly doing this.

will look, thanks.

> 
>> Thanks,
>> Michal
>>
>>
>> For 2.6.37.6
>> diff --git a/net/core/pktgen.c b/net/core/pktgen.c
>> index 33bc382..3429eb3 100644
>> --- a/net/core/pktgen.c
>> +++ b/net/core/pktgen.c
>> @@ -3500,6 +3500,13 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
>>                  pkt_dev->last_pkt_size = pkt_dev->skb->len;
>>                  pkt_dev->allocated_skbs++;
>>                  pkt_dev->clone_count = 0;       /* reset counter */
>> +       } else {
>> +               struct iphdr *iph;
>> +               iph = ip_hdr(pkt_dev->skb);
>> +               iph->id = htons(pkt_dev->ip_id);
>> +               pkt_dev->ip_id++;
>> +               iph->check = 0;
>> +               iph->check = ip_fast_csum((void *)iph, iph->ihl);
>>          }
>>
>>          if (pkt_dev->delay && pkt_dev->last_ok)
>>
>>
>>
> 
> 
> Well, you cant do that in pktgen, since you're changing previous packet
> content (it might still be in device TX queue, not yet sent, or being
> sent right now)

It is likely happening.

> 
> Now, if all you want to do is send many packets from pktgen (with only
> ID changing), you could add a fast path to not rebuild from scratch new
> packets.

What do you mean?

Thanks,
Michal


-- 
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian

Re: Hight speed data sending from custom IP out of kernel

[Eric Dumazet]

Le lundi 25 avril 2011 à 14:18 +0200, Michal Simek a écrit :

> > 
> > Now, if all you want to do is send many packets from pktgen (with only
> > ID changing), you could add a fast path to not rebuild from scratch new
> > packets.
> 
> What do you mean?


If you know your device has X slots in its TX ring buffer, you would
have to maintain at least X+1 skbs in pktgen to make sure you reuse an
skb while its previous logical content was sent on wire.

Then you are free to only change iph->id and iph->check very fast.

Re: Hight speed data sending from custom IP out of kernel

[Eric Dumazet]

Le lundi 25 avril 2011 à 14:27 +0200, Eric Dumazet a écrit :
> Le lundi 25 avril 2011 à 14:18 +0200, Michal Simek a écrit :
> 
> > > 
> > > Now, if all you want to do is send many packets from pktgen (with only
> > > ID changing), you could add a fast path to not rebuild from scratch new
> > > packets.
> > 
> > What do you mean?
> 
> 
> If you know your device has X slots in its TX ring buffer, you would
> have to maintain at least X+1 skbs in pktgen to make sure you reuse an
> skb while its previous logical content was sent on wire.
> 
> Then you are free to only change iph->id and iph->check very fast.
> 

Checking skb->users would also be a good way to know if TX completion
released skb reference. If your module owns the last reference, it can
do a recycle.

Re: Hight speed data sending from custom IP out of kernel

[Michal Simek]

Eric Dumazet wrote:
> Le lundi 25 avril 2011 à 14:27 +0200, Eric Dumazet a écrit :
>> Le lundi 25 avril 2011 à 14:18 +0200, Michal Simek a écrit :
>>
>>>> Now, if all you want to do is send many packets from pktgen (with only
>>>> ID changing), you could add a fast path to not rebuild from scratch new
>>>> packets.
>>> What do you mean?
>>
>> If you know your device has X slots in its TX ring buffer, you would
>> have to maintain at least X+1 skbs in pktgen to make sure you reuse an
>> skb while its previous logical content was sent on wire.
>>
>> Then you are free to only change iph->id and iph->check very fast.
>>

got it. There is an option to setup number of BDs in the driver where I need to 
use half skb because of nr_frags=2 where two BDs are used.

> 
> Checking skb->users would also be a good way to know if TX completion
> released skb reference. If your module owns the last reference, it can
> do a recycle.

Ok. I see that dev_kfree_skb(consume_skb).

Thanks will try,
Michal

-- 
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian

Re: Hight speed data sending from custom IP out of kernel

[zhou rui]

On Wed, Apr 20, 2011 at 12:02 AM, juice <juice@swagman.org> wrote:
>
> Hi!
>
> I can see you are probably going to run into CPU performance problems, but
> it depends a lot on the type of traffic you are going to send.
>
> My system requires quite fast processor, but even more important is to
> have a network interface card that really supports the full speed of
> gigabit ethernet line. The reason for that is that my test traffic
> includes streams of very small packets that cause a lot of overhead in
> processing.
>
> Most of my test traffic is UDP, but it does not really matter what the
> higher layers of the traffic are, this scheme operates on the ethernet
> layer and does not care about payload structure.
>
> I tried several NIC:s before i settled using Intel 82576 cards with the
> igb driver. If you have less capable interface card, your small packet
> performance is going to be a lot poorer.
>
> Using that card I can get to full speed GE line rate even with 64byte
> packets, but if you want to send larger packets, say close to 1500byte
> then almost any NIC will work OK for you.
>
> You can download the module code and the userland seeding application from
> my svn server at https://toosa.swagman.org/svn/streamgen
> The streamseed userland application requires libpcap-dev to build
> correctly but the streamgen module is self-sufficent.
>
> There is not a lot of documentation, and the module is still "work in
> progress" as I am going to fix it to work with more than one interface at
> the same time when I get to do it. Currently it can only use one interface
> on the sending host machine.
>
>  - Juice -
>

does it use the same command/config file as pktgen?
or special command?



>
>> Hi Juice,
>>
>> juice wrote:
>>> Hi Michal.
>>>
>>> How fast do you need to send the data?
>>
>> It sounds weird but as fast as possible. There is no specific limit
>> because I
>> want to create demo and test it on various hw configuration which I can
>> easily
>> create on FPGA. For now the bottleneck is Microblaze cpu. It can run from
>> 50MHz
>> till 170-180MHz. We also support both endians and have two hw IP
>> cores(10/100/1000) which I can use.
>>
>>> I have an application where I send test stream out to GE line and can
>>> fill
>>> the total capacity of the ethernet regardless of the packet size.
>>
>> What cpu do you use?
>>
>>>
>>> The test stream I am sending is stored in kernel memory, and therefore
>>> is
>>> limited by the amount of free memory. 200M is no problem.
>>
>> Is it UDP or TCP?
>>
>>>
>>> The solution I am using is loosely based on the pktgen module, except
>>> that
>>> my module can load a wireshark capture from userland program and then
>>> send
>>> it from ethernet interface in wire speed.
>>
>> Sound good. Would it be possible to see it and test it?
>>
>> Thanks,
>> Michal
>>
>>
>>>
>>>   - Juice -
>>>
>>>
>>>> Hi,
>>>> I would like to create demo for high speed data sending from custom IP
>>> through
>>>> the ethernet. I think the best description is that there are dmaable
>>>> memory
>>>> mapped registers or just memory which store data I want to send (for
>>> example 200MB).
>>>> Linux should handle all communication between target(probably server)
>>> and
>>>> host
>>>> (client) but data in the packets should go from that custom IP and
>>>> can't go
>>>> through the kernel because of performance issue.
>>>> Ethernet core have own DMA which I could use but the question is if
>>> there
>>>> is any
>>>> option how to convince the kernel that data will go directly from
>>>> memory
>>> mapped
>>>> registers and the kernel/driver/... just setup dma BD for headers and
>>> second for
>>>> data.
>>>> Do you have any experience with any solution with passing data
>>> completely
>>>> out of
>>>> kernel?
>>>> Thanks,
>>>> Michal
>>>> --
>>>> Michal Simek, Ing. (M.Eng)
>>>> w: www.monstr.eu p: +42-0-721842854
>>>> Maintainer of Linux kernel 2.6 Microblaze Linux -
>>>> http://www.monstr.eu/fdt/
>>>> Microblaze U-BOOT custodian
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>>> the
>>> body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Michal Simek, Ing. (M.Eng)
>> w: www.monstr.eu p: +42-0-721842854
>> Maintainer of Linux kernel 2.6 Microblaze Linux -
>> http://www.monstr.eu/fdt/
>> Microblaze U-BOOT custodian
>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Hight speed data sending from custom IP out of kernel

[zhou rui]

On Wed, Apr 20, 2011 at 12:02 AM, juice <juice@swagman.org> wrote:
>
> Hi!
>
> I can see you are probably going to run into CPU performance problems, but
> it depends a lot on the type of traffic you are going to send.
>
> My system requires quite fast processor, but even more important is to
> have a network interface card that really supports the full speed of
> gigabit ethernet line. The reason for that is that my test traffic
> includes streams of very small packets that cause a lot of overhead in
> processing.
>
> Most of my test traffic is UDP, but it does not really matter what the
> higher layers of the traffic are, this scheme operates on the ethernet
> layer and does not care about payload structure.
>
> I tried several NIC:s before i settled using Intel 82576 cards with the
> igb driver. If you have less capable interface card, your small packet
> performance is going to be a lot poorer.
>
> Using that card I can get to full speed GE line rate even with 64byte
> packets, but if you want to send larger packets, say close to 1500byte
> then almost any NIC will work OK for you.
>
> You can download the module code and the userland seeding application from
> my svn server at https://toosa.swagman.org/svn/streamgen
> The streamseed userland application requires libpcap-dev to build
> correctly but the streamgen module is self-sufficent.
>
> There is not a lot of documentation, and the module is still "work in
> progress" as I am going to fix it to work with more than one interface at
> the same time when I get to do it. Currently it can only use one interface
> on the sending host machine.
>
>  - Juice -
>

does it use the same command/config file as pktgen?
or special command?

 rui

>
>> Hi Juice,
>>
>> juice wrote:
>>> Hi Michal.
>>>
>>> How fast do you need to send the data?
>>
>> It sounds weird but as fast as possible. There is no specific limit
>> because I
>> want to create demo and test it on various hw configuration which I can
>> easily
>> create on FPGA. For now the bottleneck is Microblaze cpu. It can run from
>> 50MHz
>> till 170-180MHz. We also support both endians and have two hw IP
>> cores(10/100/1000) which I can use.
>>
>>> I have an application where I send test stream out to GE line and can
>>> fill
>>> the total capacity of the ethernet regardless of the packet size.
>>
>> What cpu do you use?
>>
>>>
>>> The test stream I am sending is stored in kernel memory, and therefore
>>> is
>>> limited by the amount of free memory. 200M is no problem.
>>
>> Is it UDP or TCP?
>>
>>>
>>> The solution I am using is loosely based on the pktgen module, except
>>> that
>>> my module can load a wireshark capture from userland program and then
>>> send
>>> it from ethernet interface in wire speed.
>>
>> Sound good. Would it be possible to see it and test it?
>>
>> Thanks,
>> Michal
>>
>>
>>>
>>>   - Juice -
>>>
>>>
>>>> Hi,
>>>> I would like to create demo for high speed data sending from custom IP
>>> through
>>>> the ethernet. I think the best description is that there are dmaable
>>>> memory
>>>> mapped registers or just memory which store data I want to send (for
>>> example 200MB).
>>>> Linux should handle all communication between target(probably server)
>>> and
>>>> host
>>>> (client) but data in the packets should go from that custom IP and
>>>> can't go
>>>> through the kernel because of performance issue.
>>>> Ethernet core have own DMA which I could use but the question is if
>>> there
>>>> is any
>>>> option how to convince the kernel that data will go directly from
>>>> memory
>>> mapped
>>>> registers and the kernel/driver/... just setup dma BD for headers and
>>> second for
>>>> data.
>>>> Do you have any experience with any solution with passing data
>>> completely
>>>> out of
>>>> kernel?
>>>> Thanks,
>>>> Michal
>>>> --
>>>> Michal Simek, Ing. (M.Eng)
>>>> w: www.monstr.eu p: +42-0-721842854
>>>> Maintainer of Linux kernel 2.6 Microblaze Linux -
>>>> http://www.monstr.eu/fdt/
>>>> Microblaze U-BOOT custodian
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>>> the
>>> body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Michal Simek, Ing. (M.Eng)
>> w: www.monstr.eu p: +42-0-721842854
>> Maintainer of Linux kernel 2.6 Microblaze Linux -
>> http://www.monstr.eu/fdt/
>> Microblaze U-BOOT custodian
>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Hight speed data sending from custom IP out of kernel

[juice]

>> There is not a lot of documentation, and the module is still "work in
>> progress" as I am going to fix it to work with more than one interface
>> at
>> the same time when I get to do it. Currently it can only use one
>> interface
>> on the sending host machine.
>>
>
> does it use the same command/config file as pktgen?
> or special command?
>

The commands are similar type but different than pktgen.

First, when loading the module the intreface to be used must be given as
parameter, like this: "sudo insmod ./streamgen.ko interface=eth0"
Optional parameter is the UDP port that the userspace loader program uses
to communicate with the module. default port is 5555.

Then, the streamer module creates file "/proc/net/streamgen/control" which
is used to read status of module and command it:

juice@alaspin:~$ sudo cat /proc/net/streamgen/control
Stream Generator v.0.02

  State:      off
  Repeat:     1
  Queue size: 600 packets

juice@alaspin:~$

The commands that can be used are "repeat <num>", "start", "stop",
"delete" and "clear_counters"
The repeat command sets how many times the loaded sequence is output, zero
means forever. Commands run and stop work as expected. Command delete
clears the sequence from memory and command clear_counters zeroes the
packet counters.