Packet reordering in pcap capture file

Packet reordering in pcap capture file

[Alan Shieh]

Hi everyone,

I sometimes see packets stored out of order in pcap files that generated 
by "tcpdump -i any" on kernel 2.4.26 with all packets arriving and 
departing on an e1000 NIC. That is, the ordering by receive timestamp on 
the packets is not the same as the ordering of the packets within the file.

In my precise scenario, packets of RX packets show up in the log 230 ms 
later than they ought to based on the receive timestamp. The kernel 
behavior (e.g., the packets that are sent by this node) seems to reflect 
the arrival of the Rx packet at the position in the logfile, rather than 
the arrival time according to the timestamp.

What are some of the known causes of this behavior? I'd like to know 
what locks, etc. might be causing this processing / capture delay.
--
Alan Shieh
PhD Student, Computer Science
331 Upson Hall
Ithaca, NY 14853

Re: Packet reordering in pcap capture file

[Stephen Hemminger]

On Sat, 05 Aug 2006 03:28:38 -0400
Alan Shieh <ashieh@cs.cornell.edu> wrote:

> Hi everyone,
> 
> I sometimes see packets stored out of order in pcap files that generated 
> by "tcpdump -i any" on kernel 2.4.26 with all packets arriving and 
> departing on an e1000 NIC. That is, the ordering by receive timestamp on 
> the packets is not the same as the ordering of the packets within the file.
> 
> In my precise scenario, packets of RX packets show up in the log 230 ms 
> later than they ought to based on the receive timestamp. The kernel 
> behavior (e.g., the packets that are sent by this node) seems to reflect 
> the arrival of the Rx packet at the position in the logfile, rather than 
> the arrival time according to the timestamp.
> 
> What are some of the known causes of this behavior? I'd like to know 
> what locks, etc. might be causing this processing / capture delay.

SMP or single CPU? What is the clock source being used?
If you had a CPU like dual-core AMD that doesn't sync TSC's and
that was the clock source, the timestamps could be wrong.

Re: Packet reordering in pcap capture file

[Alan Shieh]

Stephen Hemminger wrote:
> On Sat, 05 Aug 2006 03:28:38 -0400
> Alan Shieh <ashieh@cs.cornell.edu> wrote:
> 
> 
>>Hi everyone,
>>
>>I sometimes see packets stored out of order in pcap files that generated 
>>by "tcpdump -i any" on kernel 2.4.26 with all packets arriving and 
>>departing on an e1000 NIC. That is, the ordering by receive timestamp on 
>>the packets is not the same as the ordering of the packets within the file.
>>
>>In my precise scenario, packets of RX packets show up in the log 230 ms 
>>later than they ought to based on the receive timestamp. The kernel 
>>behavior (e.g., the packets that are sent by this node) seems to reflect 
>>the arrival of the Rx packet at the position in the logfile, rather than 
>>the arrival time according to the timestamp.
>>
>>What are some of the known causes of this behavior? I'd like to know 
>>what locks, etc. might be causing this processing / capture delay.
> 
> 
> SMP or single CPU? What is the clock source being used?
> If you had a CPU like dual-core AMD that doesn't sync TSC's and
> that was the clock source, the timestamps could be wrong.

Single CPU, using TSC. The behavior of the system is as if the RTT is 
230ms, so I think a queue is building up somewhere within the kernel. I 
am trying to narrow down the possible ways my experimental code could 
have caused such a queue backlog. I've tried setting netdev->quota in 
the e1000 module to a much larger value, thus forcing the backlog to be 
processed faster, but that does not help.

Alan