[rfc] new sk_buff member: hwstamp

[rfc] new sk_buff member: hwstamp

[Octavian Purdila]

Hi everybody,

In our Linux port we added a new sk_buff member, hwstamp, to store a hardware 
generated timestamp for received packets. The hardware can also insert  
hardware timestamps in the payload on the TX path and can have the RX / TX 
clocks synchronized. We use this infrastructure to compute one-way delays 
with high precision (20ns resolution).

I know that adding new members to such heavily used structures like the skb is 
viewed harshly - especially for such exotic cases as the one described above. 
So, would such a patch be considered for inclusion? Or maybe there is a 
better way of doing the above?

Thanks,
tavi

Re: [rfc] new sk_buff member: hwstamp

[Andi Kleen]

Octavian Purdila <opurdila@ixiacom.com> writes:

> So, would such a patch be considered for inclusion? Or maybe there is a 
> better way of doing the above?

You only need this between the driver and the socket recvmsg(), don't you?

One possible alternative (I admit I haven't thought all the implications
through) would be to use a second magic internal skb for this which has the 
same UDP header, but as only payload the time stamp. Disadvantage would
be the requirement to do header parsing in the driver, but often
hardware does that already.

-Andi

Re: [rfc] new sk_buff member: hwstamp

[Oliver Hartkopp]

Andi Kleen wrote:
> Octavian Purdila <opurdila@ixiacom.com> writes:
>
>   
>> So, would such a patch be considered for inclusion? Or maybe there is a 
>> better way of doing the above?
>>     
>
> You only need this between the driver and the socket recvmsg(), don't you?
>
> One possible alternative (I admit I haven't thought all the implications
> through) would be to use a second magic internal skb for this which has the 
> same UDP header, but as only payload the time stamp. Disadvantage would
> be the requirement to do header parsing in the driver, but often
> hardware does that already.
>
>   

The additional hardware timestamp would not only be interesting for
people using UDP. There had been several discussions for CAN controllers
(controller area network) in the past, as some of these controllers also
provide HW timestamps that are important for high quality logging tools.

However i feel, that *one* nanosec resolution timestamp (as it already
exists inside the skbuff) is enough. AFAIK the timestamp is only set in
the netif_rx(), when it is not already set by the driver itself. For
that reason i would suggest to create some semi-intelligent offset
calculation inside the driver that makes the skb->tstamp value
correspond with the hw timestamp and therefore transports the high
resolution timestamp requirement into the userspace.

Regards,
Oliver

Re: [rfc] new sk_buff member: hwstamp

[Octavian Purdila]

>
> However i feel, that *one* nanosec resolution timestamp (as it already
> exists inside the skbuff) is enough. AFAIK the timestamp is only set in
> the netif_rx(), when it is not already set by the driver itself. For
> that reason i would suggest to create some semi-intelligent offset
> calculation inside the driver that makes the skb->tstamp value
> correspond with the hw timestamp and therefore transports the high
> resolution timestamp requirement into the userspace.
>

You mean something like here [1] ?

The problem for this approach in our usecase (one way delay measurement) is 
that the real time clock of the CPUs (of the RX and TX ports) is not 
synchronized, but the hw timestamp (implemented in the NIC/FPGA) is. 

[1] 
http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.23-rc4/2.6.23-rc4-mm1/broken-out/sky2-hardware-receive-timestamp-counter.patch

Thanks,
tavi

Re: [rfc] new sk_buff member: hwstamp

[Oliver Hartkopp]

Octavian Purdila wrote:
>> However i feel, that *one* nanosec resolution timestamp (as it already
>> exists inside the skbuff) is enough. AFAIK the timestamp is only set in
>> the netif_rx(), when it is not already set by the driver itself. For
>> that reason i would suggest to create some semi-intelligent offset
>> calculation inside the driver that makes the skb->tstamp value
>> correspond with the hw timestamp and therefore transports the high
>> resolution timestamp requirement into the userspace.
>>
>>     
>
> You mean something like here [1] ?
> [1] 
> http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.23-rc4/2.6.23-rc4-mm1/broken-out/sky2-hardware-receive-timestamp-counter.patch
>
>   

Yes - exactly :-)

> The problem for this approach in our usecase (one way delay measurement) is 
> that the real time clock of the CPUs (of the RX and TX ports) is not 
> synchronized, but the hw timestamp (implemented in the NIC/FPGA) is. 
>   

Is it necessary to have syncronized clocks on the different CPUs or is
it feasible to calculate a per-cpu-clock offset in your driver, so that
you can build a complete timing overview of your setup?

Regards,
Oliver

Re: [rfc] new sk_buff member: hwstamp

[Octavian Purdila]

On Monday 14 July 2008, Oliver Hartkopp wrote:
>
> > The problem for this approach in our usecase (one way delay measurement)
> > is that the real time clock of the CPUs (of the RX and TX ports) is not
> > synchronized, but the hw timestamp (implemented in the NIC/FPGA) is.
>
> Is it necessary to have syncronized clocks on the different CPUs or is
> it feasible to calculate a per-cpu-clock offset in your driver, so that
> you can build a complete timing overview of your setup?
>

Sorry,  what I meant by CPU is a complete and independent system: CPU + memory 
+ NIC. The real time of the CPUs of the two systems are not synchronized, but 
the NIC clock of both systems is.

Thanks,
tavi

Re: [rfc] new sk_buff member: hwstamp

[Octavian Purdila]

On Monday 14 July 2008, Andi Kleen wrote:

>
> You only need this between the driver and the socket recvmsg(), don't you?
>

Correct.

> One possible alternative (I admit I haven't thought all the implications
> through) would be to use a second magic internal skb for this which has the
> same UDP header, but as only payload the time stamp. Disadvantage would
> be the requirement to do header parsing in the driver, but often
> hardware does that already.
>

Hmm, this sounds interesting. 

Maybe only keep the timestamp without the UDP header and queue the skb 
as "ancillary" data?

And we can probably do this in a generic way for all sort of socket types 
(PF_PACKET comes to mind) - just add a special ancillary data queue in the 
socket. Sure this would mean stuffing the sockets structures - not very 
different from adding new stuff in skb :), but I feel it is a bit more 
generic: an easy to extend channel of communication between drivers/hw and 
userspace without future structures overhead. 

And I think this will help with solving the hw TX stamp problem I am looking 
into, as well (how to send the hw generated TX stamp to application). 

Thanks,
tavi

Re: [rfc] new sk_buff member: hwstamp

[Stephen Hemminger]

On Mon, 14 Jul 2008 18:43:00 +0300
Octavian Purdila <opurdila@ixiacom.com> wrote:

> 
> Hi everybody,
> 
> In our Linux port we added a new sk_buff member, hwstamp, to store a hardware 
> generated timestamp for received packets. The hardware can also insert  
> hardware timestamps in the payload on the TX path and can have the RX / TX 
> clocks synchronized. We use this infrastructure to compute one-way delays 
> with high precision (20ns resolution).
> 
> I know that adding new members to such heavily used structures like the skb is 
> viewed harshly - especially for such exotic cases as the one described above. 
> So, would such a patch be considered for inclusion? Or maybe there is a 
> better way of doing the above?
> 

Why not do some clock synchronization and use existing absolute timestamp?
This would benefit system generally since it would give more accurate values
for network traffic analysis.

Re: [rfc] new sk_buff member: hwstamp

[Octavian Purdila]

On Tuesday 15 July 2008, Stephen Hemminger wrote:

>
> Why not do some clock synchronization and use existing absolute timestamp?
> This would benefit system generally since it would give more accurate
> values for network traffic analysis.
>

Sorry if I repeat myself, but I get the feeling that I did a poor job 
explaining the issue, so here is another try :)

Yes, we could try to do some real-time clock synchronization, but I don't 
think that we will be able to achieve the same level of synchronization 
accuracy that we can have at the NIC level (where our timestamping units are 
hardware synchronized within 20ns).

Thanks,
tavi