What ixgbe devices support HWTSTAMP_FILTER_ALL for hardware time stamping?

What ixgbe devices support HWTSTAMP_FILTER_ALL for hardware time stamping?

[Guy Harris]

libpcap offers the ability to request hardware time stamping for packets and to inquire which forms of hardware time stamping, if any, are supported for an interface.

The Linux implementation currently implements the inquiry by doing a ETHTOOL_GET_TS_INFO SIOETHTOOL ioctl and looking at the so_timestamping bits, if the linux/ethtool.h header defines ETHTOOL_GET_TS_INFO and the ioctl succeeds on the device.

This is inadequate - as libpcap requests hardware time stamping for all packets, it should also check whether HWTSTAMP_FILTER_ALL is set in rx_filters, and only offer hardware time stamping if it's set.

The code in ixgbe_ptp.c does:

	case HWTSTAMP_FILTER_PTP_V1_L4_EVENT:
	case HWTSTAMP_FILTER_ALL:
		/* The X550 controller is capable of timestamping all packets,
		 * which allows it to accept any filter.
		 */
		if (hw->mac.type >= ixgbe_mac_X550) {
			tsync_rx_ctl |= IXGBE_TSYNCRXCTL_TYPE_ALL;
			config->rx_filter = HWTSTAMP_FILTER_ALL;
			adapter->flags |= IXGBE_FLAG_RX_HWTSTAMP_ENABLED;
			break;
		}
		/* fall through */
	default:
		/*
		 * register RXMTRL must be set in order to do V1 packets,
		 * therefore it is not possible to time stamp both V1 Sync and
		 * Delay_Req messages and hardware does not support
		 * timestamping all packets => return error
		 */
		adapter->flags &= ~(IXGBE_FLAG_RX_HWTSTAMP_ENABLED |
				    IXGBE_FLAG_RX_HWTSTAMP_IN_REGISTER);
		config->rx_filter = HWTSTAMP_FILTER_NONE;
		return -ERANGE;

which seems to indicate that only the X550 controller supports time stamping all packets in hardware.

However, the code in ixgbe_ethtool.c does:

	switch (adapter->hw.mac.type) {
	case ixgbe_mac_X550:
	case ixgbe_mac_X550EM_x:
	case ixgbe_mac_X540:
	case ixgbe_mac_82599EB:
		info->so_timestamping =
			SOF_TIMESTAMPING_TX_SOFTWARE |
			SOF_TIMESTAMPING_RX_SOFTWARE |
			SOF_TIMESTAMPING_SOFTWARE |
			SOF_TIMESTAMPING_TX_HARDWARE |
			SOF_TIMESTAMPING_RX_HARDWARE |
			SOF_TIMESTAMPING_RAW_HARDWARE;

		if (adapter->ptp_clock)
			info->phc_index = ptp_clock_index(adapter->ptp_clock);
		else
			info->phc_index = -1;

		info->tx_types =
			(1 << HWTSTAMP_TX_OFF) |
			(1 << HWTSTAMP_TX_ON);

		info->rx_filters =
			(1 << HWTSTAMP_FILTER_NONE) |
			(1 << HWTSTAMP_FILTER_PTP_V1_L4_SYNC) |
			(1 << HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ) |
			(1 << HWTSTAMP_FILTER_PTP_V2_EVENT);
		break;
	default:
		return ethtool_op_get_ts_info(dev, info);
	}

which draws no distinction between the X550 controller and the X540 and 82599, and doesn't say *any* of them support HWTSTAMP_FILTER_ALL.

Is it the case that only the ixgbe_mac_X550 and ixgbe_mac_X550EM_x controllers support HWTSTAMP_FILTER_ALL?  If so, shouldn't ixgbe_get_ts_info() be doing something such as:

	switch (adapter->hw.mac.type) {
	case ixgbe_mac_X550:
	case ixgbe_mac_X550EM_x:
	case ixgbe_mac_X540:
	case ixgbe_mac_82599EB:
		info->so_timestamping =
			SOF_TIMESTAMPING_TX_SOFTWARE |
			SOF_TIMESTAMPING_RX_SOFTWARE |
			SOF_TIMESTAMPING_SOFTWARE |
			SOF_TIMESTAMPING_TX_HARDWARE |
			SOF_TIMESTAMPING_RX_HARDWARE |
			SOF_TIMESTAMPING_RAW_HARDWARE;

		if (adapter->ptp_clock)
			info->phc_index = ptp_clock_index(adapter->ptp_clock);
		else
			info->phc_index = -1;

		info->tx_types =
			(1 << HWTSTAMP_TX_OFF) |
			(1 << HWTSTAMP_TX_ON);

		info->rx_filters =
			(1 << HWTSTAMP_FILTER_NONE) |
			(1 << HWTSTAMP_FILTER_PTP_V1_L4_SYNC) |
			(1 << HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ) |
			(1 << HWTSTAMP_FILTER_PTP_V2_EVENT);
		if (adapter->hw.mac.type >= ixgbe_mac_X550)
			info->rx_filters |= (1 << HWTSTAMP_FILTER_ALL);
		break;
	default:
		return ethtool_op_get_ts_info(dev, info);
	}

>From a quick scan of drivers/net, it looks as if

	drivers/net/ethernet/cavium/liquidio

also support HWTSTAMP_FILTER_ALL but don't advertise it, 

	drivers/net/ethernet/chelsio/cxgb4

doesn't set rx_filters (or tx_types) and thus doesn't appear to advertise support for *any* filters, 

	drivers/net/ethernet/freescale/gianfar*.c

reports HWTSTAMP_FILTER_ALL even if the device doesn't have FSL_GIANFAR_DEV_HAS_TIMER set so that it claims that devices without a timer can do hardware timestamping,

	drivers/net/ethernet/stmicro/stmmac/

appears to advertise a whole bunch of HWTSTAMP_FILTER_ types regardless of whether the device supports "IEEE 1588-2008 Advanced Time Stamp", but, if it doesn't support that, it maps all supports filters other than HWTSTAMP_FILTER_NONE to HWTSTAMP_FILTER_PTP_V1_L4_EVENT (which is misleading), and

	drivers/net/ethernet/cavium/octeon

	drivers/net/ethernet/neterion/vxge/

	drivers/net/ethernet/tile/tilegx.c

don't even provide their own get_ts_info routines and thus doesn't advertise hardware timestamping at all.

Re: What ixgbe devices support HWTSTAMP_FILTER_ALL for hardware time stamping?

[Guy Harris]

On May 13, 2016, at 4:12 PM, Guy Harris <guy@alum.mit.edu> wrote:

> 	drivers/net/ethernet/freescale/gianfar*.c
> 
> reports HWTSTAMP_FILTER_ALL even if the device doesn't have FSL_GIANFAR_DEV_HAS_TIMER set so that it claims that devices without a timer can do hardware timestamping,

Nope - I checked, and it doesn't claim to support any hardware timestamping at all if it doesn't have FSL_GIANFAR_DEV_HAS_TIMER set, so that *particular* driver is OK.

Re: What ixgbe devices support HWTSTAMP_FILTER_ALL for hardware time stamping?

[Richard Cochran]

On Fri, May 13, 2016 at 04:12:52PM -0700, Guy Harris wrote:
> The Linux implementation currently implements the inquiry by doing a
> ETHTOOL_GET_TS_INFO SIOETHTOOL ioctl and looking at the
> so_timestamping bits, if the linux/ethtool.h header defines
> ETHTOOL_GET_TS_INFO and the ioctl succeeds on the device.

So far, so good. 

> This is inadequate - as libpcap requests hardware time stamping for
> all packets, it should also check whether HWTSTAMP_FILTER_ALL is set
> in rx_filters, and only offer hardware time stamping if it's set.

The SO_TIMESTAMPING and SIOCSHWTSTAMP interfaces predate
ETHTOOL_GET_TS_INFO, and they work fine without it.  Applications
should simply use SIOCSHWTSTAMP to request the mode that they need and
check the result.

That said, the information in ETHTOOL_GET_TS_INFO should be correct.

> Is it the case that only the ixgbe_mac_X550 and ixgbe_mac_X550EM_x
> controllers support HWTSTAMP_FILTER_ALL?  

Looks like it.

> If so, shouldn't ixgbe_get_ts_info() be doing something such as:

> 		if (adapter->hw.mac.type >= ixgbe_mac_X550)
> 			info->rx_filters |= (1 << HWTSTAMP_FILTER_ALL);

Yes, probably.

> From a quick scan of drivers/net, it looks as if
> 
> 	drivers/net/ethernet/cavium/liquidio
> 
> also support HWTSTAMP_FILTER_ALL but don't advertise it, 

For this and the other drivers you mentioned, their maintainers might
appreciate patches...

Thanks,
Richard

Re: What ixgbe devices support HWTSTAMP_FILTER_ALL for hardware time stamping?

[Jeff Kirsher]

[-- Attachment #1: Type: text/plain, Size: 6745 bytes --]

On Fri, 2016-05-13 at 16:12 -0700, Guy Harris wrote:
> libpcap offers the ability to request hardware time stamping for packets
> and to inquire which forms of hardware time stamping, if any, are
> supported for an interface.
> 
> The Linux implementation currently implements the inquiry by doing a
> ETHTOOL_GET_TS_INFO SIOETHTOOL ioctl and looking at the so_timestamping
> bits, if the linux/ethtool.h header defines ETHTOOL_GET_TS_INFO and the
> ioctl succeeds on the device.
> 
> This is inadequate - as libpcap requests hardware time stamping for all
> packets, it should also check whether HWTSTAMP_FILTER_ALL is set in
> rx_filters, and only offer hardware time stamping if it's set.
> 
> The code in ixgbe_ptp.c does:
> 
>         case HWTSTAMP_FILTER_PTP_V1_L4_EVENT:
>         case HWTSTAMP_FILTER_ALL:
>                 /* The X550 controller is capable of timestamping all
> packets,
>                  * which allows it to accept any filter.
>                  */
>                 if (hw->mac.type >= ixgbe_mac_X550) {
>                         tsync_rx_ctl |= IXGBE_TSYNCRXCTL_TYPE_ALL;
>                         config->rx_filter = HWTSTAMP_FILTER_ALL;
>                         adapter->flags |= IXGBE_FLAG_RX_HWTSTAMP_ENABLED;
>                         break;
>                 }
>                 /* fall through */
>         default:
>                 /*
>                  * register RXMTRL must be set in order to do V1 packets,
>                  * therefore it is not possible to time stamp both V1
> Sync and
>                  * Delay_Req messages and hardware does not support
>                  * timestamping all packets => return error
>                  */
>                 adapter->flags &= ~(IXGBE_FLAG_RX_HWTSTAMP_ENABLED |
>                                     IXGBE_FLAG_RX_HWTSTAMP_IN_REGISTER);
>                 config->rx_filter = HWTSTAMP_FILTER_NONE;
>                 return -ERANGE;
> 
> which seems to indicate that only the X550 controller supports time
> stamping all packets in hardware.
> 
> However, the code in ixgbe_ethtool.c does:
> 
>         switch (adapter->hw.mac.type) {
>         case ixgbe_mac_X550:
>         case ixgbe_mac_X550EM_x:
>         case ixgbe_mac_X540:
>         case ixgbe_mac_82599EB:
>                 info->so_timestamping =
>                         SOF_TIMESTAMPING_TX_SOFTWARE |
>                         SOF_TIMESTAMPING_RX_SOFTWARE |
>                         SOF_TIMESTAMPING_SOFTWARE |
>                         SOF_TIMESTAMPING_TX_HARDWARE |
>                         SOF_TIMESTAMPING_RX_HARDWARE |
>                         SOF_TIMESTAMPING_RAW_HARDWARE;
> 
>                 if (adapter->ptp_clock)
>                         info->phc_index = ptp_clock_index(adapter-
> >ptp_clock);
>                 else
>                         info->phc_index = -1;
> 
>                 info->tx_types =
>                         (1 << HWTSTAMP_TX_OFF) |
>                         (1 << HWTSTAMP_TX_ON);
> 
>                 info->rx_filters =
>                         (1 << HWTSTAMP_FILTER_NONE) |
>                         (1 << HWTSTAMP_FILTER_PTP_V1_L4_SYNC) |
>                         (1 << HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ) |
>                         (1 << HWTSTAMP_FILTER_PTP_V2_EVENT);
>                 break;
>         default:
>                 return ethtool_op_get_ts_info(dev, info);
>         }
> 
> which draws no distinction between the X550 controller and the X540 and
> 82599, and doesn't say *any* of them support HWTSTAMP_FILTER_ALL.
> 
> Is it the case that only the ixgbe_mac_X550 and ixgbe_mac_X550EM_x
> controllers support HWTSTAMP_FILTER_ALL?  If so, shouldn't
> ixgbe_get_ts_info() be doing something such as:
> 
>         switch (adapter->hw.mac.type) {
>         case ixgbe_mac_X550:
>         case ixgbe_mac_X550EM_x:
>         case ixgbe_mac_X540:
>         case ixgbe_mac_82599EB:
>                 info->so_timestamping =
>                         SOF_TIMESTAMPING_TX_SOFTWARE |
>                         SOF_TIMESTAMPING_RX_SOFTWARE |
>                         SOF_TIMESTAMPING_SOFTWARE |
>                         SOF_TIMESTAMPING_TX_HARDWARE |
>                         SOF_TIMESTAMPING_RX_HARDWARE |
>                         SOF_TIMESTAMPING_RAW_HARDWARE;
> 
>                 if (adapter->ptp_clock)
>                         info->phc_index = ptp_clock_index(adapter-
> >ptp_clock);
>                 else
>                         info->phc_index = -1;
> 
>                 info->tx_types =
>                         (1 << HWTSTAMP_TX_OFF) |
>                         (1 << HWTSTAMP_TX_ON);
> 
>                 info->rx_filters =
>                         (1 << HWTSTAMP_FILTER_NONE) |
>                         (1 << HWTSTAMP_FILTER_PTP_V1_L4_SYNC) |
>                         (1 << HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ) |
>                         (1 << HWTSTAMP_FILTER_PTP_V2_EVENT);
>                 if (adapter->hw.mac.type >= ixgbe_mac_X550)
>                         info->rx_filters |= (1 << HWTSTAMP_FILTER_ALL);
>                 break;
>         default:
>                 return ethtool_op_get_ts_info(dev, info);
>         }

Are you planning to produce a patch or are you wanting us to do the work to
fix the issue?  Just asking so that work is not duplicated.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: What ixgbe devices support HWTSTAMP_FILTER_ALL for hardware time stamping?

[Guy Harris]

On May 14, 2016, at 12:30 AM, Richard Cochran <richardcochran@gmail.com> wrote:

> On Fri, May 13, 2016 at 04:12:52PM -0700, Guy Harris wrote:
>> The Linux implementation currently implements the inquiry by doing a
>> ETHTOOL_GET_TS_INFO SIOETHTOOL ioctl and looking at the
>> so_timestamping bits, if the linux/ethtool.h header defines
>> ETHTOOL_GET_TS_INFO and the ioctl succeeds on the device.
> 
> So far, so good. 
> 
>> This is inadequate - as libpcap requests hardware time stamping for
>> all packets, it should also check whether HWTSTAMP_FILTER_ALL is set
>> in rx_filters, and only offer hardware time stamping if it's set.
> 
> The SO_TIMESTAMPING and SIOCSHWTSTAMP interfaces predate
> ETHTOOL_GET_TS_INFO, and they work fine without it.  Applications
> should simply use SIOCSHWTSTAMP to request the mode that they need and
> check the result.

So if you have a GUI application for packet capture, with a combo box to select the type of time stamping, should it:

	1) regardless of whether ETHTOOL_GET_TS_INFO is available, open the adapter, try each of the time stamp types to see whether it works, and show a combo box based on that;

	2) use ETHTOOL_GET_TS_INFO if available;

	3) offer all possibilities regardless of whether they work with the adapter or not, and just report an error for possibilities that don't work?

My preference is 2) - which is the main reason why libpcap offers "what possibilities are available?" APIs, not just "request this possibility" APIs.

Re: What ixgbe devices support HWTSTAMP_FILTER_ALL for hardware time stamping?

[Guy Harris]

On May 14, 2016, at 12:41 AM, Jeff Kirsher <tarbal@gmail.com> wrote:

> Are you planning to produce a patch or are you wanting us to do the work to
> fix the issue?  Just asking so that work is not duplicated.

I'm willing to produce the patches, although

	1) I don't currently have a platform set up to test whether they compile;

	2) I don't have hardware on which to test whether they work (the person who submitted

		https://github.com/the-tcpdump-group/tcpdump/issues/393#issuecomment-218442072

	   does, although he doesn't have X550 hardware - I guess he wants hardware time stamping on an 82599EB; it sounds as if that won't work, and his patch to the driver won't give him what he wants);

	3) patches from the driver maintainers might 1) be more likely to do the right thing and 2) be more likely to be accepted.

Re: What ixgbe devices support HWTSTAMP_FILTER_ALL for hardware time stamping?

[Richard Cochran]

On Sat, May 14, 2016 at 11:47:22AM -0700, Guy Harris wrote:
> So if you have a GUI application for packet capture, with a combo box to select the type of time stamping, should it:
> 
> 	1) regardless of whether ETHTOOL_GET_TS_INFO is available, open the adapter, try each of the time stamp types to see whether it works, and show a combo box based on that;
> 
> 	2) use ETHTOOL_GET_TS_INFO if available;
> 
> 	3) offer all possibilities regardless of whether they work with the adapter or not, and just report an error for possibilities that don't work?
> 
> My preference is 2) - which is the main reason why libpcap offers "what possibilities are available?" APIs, not just "request this possibility" APIs.

You are going to have to implement #1 in any case, if you want your
program to work on all kernels.  IIRC get_ts_info appeared in 3.5.

Thanks,
Richard

Re: What ixgbe devices support HWTSTAMP_FILTER_ALL for hardware time stamping?

[Guy Harris]

On May 14, 2016, at 1:26 PM, Richard Cochran <richardcochran@gmail.com> wrote:

> On Sat, May 14, 2016 at 11:47:22AM -0700, Guy Harris wrote:
>> So if you have a GUI application for packet capture, with a combo box to select the type of time stamping, should it:
>> 
>> 	1) regardless of whether ETHTOOL_GET_TS_INFO is available, open the adapter, try each of the time stamp types to see whether it works, and show a combo box based on that;
>> 
>> 	2) use ETHTOOL_GET_TS_INFO if available;
>> 
>> 	3) offer all possibilities regardless of whether they work with the adapter or not, and just report an error for possibilities that don't work?
>> 
>> My preference is 2) - which is the main reason why libpcap offers "what possibilities are available?" APIs, not just "request this possibility" APIs.
> 
> You are going to have to implement #1 in any case, if you want your
> program to work on all kernels.

What libpcap currently implements is a combination of #2 and #3, where:

	if it's compiled with headers that define ETHTOOL_GET_TS_INFO, it tries to do ETHTOOL_GET_TS_INFO and, if that fails with EOPNOTSUPP or EINVAL, it offers all possibilities;

	if it's compiled with headers that don't define it, it just offers all possibilities.

It could do a combination of #2 and #1, where "offers all possibilities" is replaced by "opens the adapter, tries each of the possibilities, and offers the ones that don't fail" - but, other than the current bugs with ETHTOOL_GET_TS_INFO, I don't see any advantage to doing only #1, rather than trying #2, perhaps with some special-casing to work around the bugs in question, and only falling back on actually trying to set the options if we can't ask about them.

Re: What ixgbe devices support HWTSTAMP_FILTER_ALL for hardware time stamping?

[Richard Cochran]

On Sat, May 14, 2016 at 07:11:50PM -0700, Guy Harris wrote:
> It could do a combination of #2 and #1, where "offers all
> possibilities" is replaced by "opens the adapter, tries each of the
> possibilities, and offers the ones that don't fail" - but, other
> than the current bugs with ETHTOOL_GET_TS_INFO, I don't see any
> advantage to doing only #1, rather than trying #2, perhaps with some
> special-casing to work around the bugs in question, and only falling
> back on actually trying to set the options if we can't ask about
> them.

Right.

Regarding the drivers you found, even if you can't patch them
yourself, please write the maintainers directly with me and netdev on
CC in order to bring attention to the bugs.

The get_maintainer.pl script can help identify the maintainers.

Thanks,
Richard