Re: [RFC PATCH 04/14] mbuf: replace data pointer by an offset

[Olivier MATZ <olivier.matz-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>]

Hi Bruce,

On 08/11/2014 10:44 PM, Bruce Richardson wrote:
> From: Olivier Matz <olivier.matz-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>
>
> Original patch:
>   The mbuf structure already contains a pointer to the beginning of the
>   buffer (m->buf_addr). It is not needed to use 8 bytes again to store
>   another pointer to the beginning of the data.
>
>   Using a 16 bits unsigned integer is enough as we know that a mbuf is
>   never longer than 64KB. We gain 6 bytes in the structure thanks to
>   this modification.
>
>   Signed-off-by: Olivier Matz <olivier.matz-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>
>
> This version:
> * Updated original patch to apply to latest on mainline.
> * Disabled vector PMD in config as it relies heavily on the mbuf layout
>
> Signed-off-by: Bruce Richardson <bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Remaining references shown by:

git grep 
"\(pkt->data[^_]\)\|\(mb->data[^_]\)\|\(m->data[^_]\)\|\(mbuf->data[^_]\)"

In:
app/test-pmd/ieee1588fwd.c
examples/vhost_xen/main.c
lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
lib/librte_pmd_pcap/rte_eth_pcap.c
lib/librte_pmd_xenvirt/rte_eth_xenvirt.c

> --- a/lib/librte_mbuf/rte_mbuf.h
> +++ b/lib/librte_mbuf/rte_mbuf.h
> @@ -140,6 +140,13 @@ struct rte_mbuf {
>  	void *buf_addr;           /**< Virtual address of segment buffer. */
>  	phys_addr_t buf_physaddr; /**< Physical address of segment buffer. */
>  	uint16_t buf_len;         /**< Length of segment buffer. */
> +
> +	/* valid for any segment */
> +	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
> +	uint16_t data_off;
> +	uint16_t data_len;        /**< Amount of data in segment buffer. */
> +	uint32_t pkt_len;         /**< Total pkt len: sum of all segments. */
> +
>  #ifdef RTE_MBUF_REFCNT
>  	/**
>  	 * 16-bit Reference counter.
> @@ -156,18 +163,12 @@ struct rte_mbuf {
>  #else
>  	uint16_t refcnt_reserved;     /**< Do not use this field */
>  #endif
> -	uint16_t reserved;             /**< Unused field. Required for padding. */
> -	uint16_t ol_flags;            /**< Offload features. */
> -
> -	/* valid for any segment */
> -	struct rte_mbuf *next;  /**< Next segment of scattered packet. */
> -	void* data;             /**< Start address of data in segment buffer. */
> -	uint16_t data_len;      /**< Amount of data in segment buffer. */
>
>  	/* these fields are valid for first segment only */
>  	uint8_t nb_segs;        /**< Number of segments. */
>  	uint8_t in_port;        /**< Input port. */
> -	uint32_t pkt_len;       /**< Total pkt len: sum of all segment data_len. */
> +	uint16_t ol_flags;            /**< Offload features. */
> +	uint16_t reserved;             /**< Unused field. Required for padding. */

I think we should try to keep comments aligned if possible.

>
>  	/* offload features, valid for first segment only */
>  	union rte_vlan_macip vlan_macip;
> @@ -185,7 +186,7 @@ struct rte_mbuf {
>  		uint16_t metadata16[0];
>  		uint32_t metadata32[0];
>  		uint64_t metadata64[0];
> -	};
> +	} __rte_cache_aligned;
>  } __rte_cache_aligned;
>

In my initial patch, there was a "reserved2" field at the end of the
rte_mbuf structure to keep its size to 64 bytes. This is not really
required because of the __rte_cache_aligned, but I wonder if it's a
problem to have metadata not starting on a cache line. There can be
some conflicts if some part of the code use *(uint32 *)(m + 1)
and other part of code m->metadata32[0].

By the way (that's completely off-topic), but I don't really see why
having this metadata at the end of mbuf structure is useful.

> @@ -1523,7 +1523,8 @@ ixgbe_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
>   		}
>
>   		/* Prefetch data of first segment, if configured to do so. */
> -		rte_packet_prefetch(first_seg->data);
> +		rte_packet_prefetch((char *)first_seg->buf_addr +
> +			first_seg->data_off);
>

It seems there is a trailing whitespace here after the "+" (seen by
"git am").


Regards,
Olivier