igc: AF_PACKET and SO_TXTIME question

igc: AF_PACKET and SO_TXTIME question

[Ferenc Fejes]

Hi,

We are experimenting with scheduled packet transfers using the
AF_PACKET socket. There is the ETF disk, which is great for most cases.
When we bypassed ETF, everything seemed ok regarding the timing: our
packet received about +/-15ns offset at the receiver (now its the same
machine just to make sure with the timesync) compared to the timestamp
set with SO_TXTIME CMSG.

What we tried now is to bypass the ETF qdisc. We enabled the ETF qdisc
with hardware offload and sent the exact same packets, but this time
with PACKET_QDISC_BAYPASS enabled on the AF_PACKET socket. The codepath
looks good, the qdisc part is not called, the packet_snd calls the
dev_direct_xmit which calls the igc_xmit_frame. However, in this case
the packet was sent more or less immediately.

I wonder why we do not see the delayed sending in this case? We tried
with different offsets (e.g. 0.1, 0.01, 0.001 sec in the future) but we
received the packet after 20-30usec every time. I cant see any code
that touches the skb timestamp after the packet_snd, so I suspect that
the igc_xmit_frame sees the same timestamp that it would see in the
non-baypass case.

I happen to have the i225 user manual, but after some grep I cannot
find any debug registers or counters to monitor the behavior of the
scheduled transmission (scheduling errors or bad timestamps, etc.). Are
there any?

I am afraid this issue might also be relevant for the AF_XDP case,
which also hooks after the qdisc layer, so the launchtime (or whatever
it is called) is handled directly by the igc driver.

Best,
Ferenc

Re: igc: AF_PACKET and SO_TXTIME question

[Ferenc Fejes]

Small correction

On Wed, 2024-02-14 at 15:00 +0100, Ferenc Fejes wrote:
> Hi,
> 
> We are experimenting with scheduled packet transfers using the
> AF_PACKET socket. There is the ETF disk, which is great for most
> cases.
> When we bypassed ETF, everything seemed ok regarding the timing: our

this was meant to be "when we NOT bypassed"

> packet received about +/-15ns offset at the receiver (now its the
> same
> machine just to make sure with the timesync) compared to the
> timestamp
> set with SO_TXTIME CMSG.
> 
> What we tried now is to bypass the ETF qdisc. We enabled the ETF
> qdisc
> with hardware offload and sent the exact same packets, but this time
> with PACKET_QDISC_BAYPASS enabled on the AF_PACKET socket. The
> codepath
> looks good, the qdisc part is not called, the packet_snd calls the
> dev_direct_xmit which calls the igc_xmit_frame. However, in this case
> the packet was sent more or less immediately.
> 
> I wonder why we do not see the delayed sending in this case? We tried
> with different offsets (e.g. 0.1, 0.01, 0.001 sec in the future) but
> we
> received the packet after 20-30usec every time. I cant see any code
> that touches the skb timestamp after the packet_snd, so I suspect
> that
> the igc_xmit_frame sees the same timestamp that it would see in the
> non-baypass case.
> 
> I happen to have the i225 user manual, but after some grep I cannot
> find any debug registers or counters to monitor the behavior of the
> scheduled transmission (scheduling errors or bad timestamps, etc.).
> Are
> there any?
> 
> I am afraid this issue might also be relevant for the AF_XDP case,
> which also hooks after the qdisc layer, so the launchtime (or
> whatever
> it is called) is handled directly by the igc driver.
> 
> Best,
> Ferenc

Re: igc: AF_PACKET and SO_TXTIME question

[Kurt Kanzenbach]

[-- Attachment #1: Type: text/plain, Size: 2322 bytes --]

Hi Ferenc,

On Wed Feb 14 2024, Ferenc Fejes wrote:
> Hi,
>
> We are experimenting with scheduled packet transfers using the
> AF_PACKET socket. There is the ETF disk, which is great for most cases.

For what use cases does it not work? Are you running PREEMPT_RT? Just
asking out of curiosity :-).

> When we bypassed ETF, everything seemed ok regarding the timing: our
> packet received about +/-15ns offset at the receiver (now its the same
> machine just to make sure with the timesync) compared to the timestamp
> set with SO_TXTIME CMSG.
>
> What we tried now is to bypass the ETF qdisc. We enabled the ETF qdisc
> with hardware offload and sent the exact same packets, but this time
> with PACKET_QDISC_BAYPASS enabled on the AF_PACKET socket. The codepath
> looks good, the qdisc part is not called, the packet_snd calls the
> dev_direct_xmit which calls the igc_xmit_frame. However, in this case
> the packet was sent more or less immediately.

Well, yeah the code path looks good indeed. packet_snd() copies the
transmit time which is provided by the CMSG and calls into
packet_xmit(), dev_direct_xmit()...

>
> I wonder why we do not see the delayed sending in this case? We tried
> with different offsets (e.g. 0.1, 0.01, 0.001 sec in the future) but we
> received the packet after 20-30usec every time. I cant see any code
> that touches the skb timestamp after the packet_snd, so I suspect that
> the igc_xmit_frame sees the same timestamp that it would see in the
> non-baypass case.

Maybe add some trace_printk()s to track what timestamps are actually
calculated in igc_tx_launchtime() and if it makes sense?

Second point to make sure is that the Tx queue your packet is being
transmitted to has Launch Time enabled.

>
> I happen to have the i225 user manual, but after some grep I cannot
> find any debug registers or counters to monitor the behavior of the
> scheduled transmission (scheduling errors or bad timestamps, etc.). Are
> there any?

Not that I'm aware of.

>
> I am afraid this issue might also be relevant for the AF_XDP case,
> which also hooks after the qdisc layer, so the launchtime (or whatever
> it is called) is handled directly by the igc driver.

Is that already possible with AF_XDP? There were some patches on
xdp-hints, but i don't think it has been merged yet.

Thanks,
Kurt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: igc: AF_PACKET and SO_TXTIME question

[Ferenc Fejes]

Hi Kurt,

On Wed, 2024-02-14 at 15:51 +0100, Kurt Kanzenbach wrote:
> Hi Ferenc,
> 
> On Wed Feb 14 2024, Ferenc Fejes wrote:
> > Hi,
> > 
> > We are experimenting with scheduled packet transfers using the
> > AF_PACKET socket. There is the ETF disk, which is great for most
> > cases.
> 
> For what use cases does it not work? Are you running PREEMPT_RT? Just
> asking out of curiosity :-).

For some experiment, I need control over when the packets drained from
the ETF queue to the NIC. Right now if you schedule some packets lets
say 0.5 sec in the future, they are sitting inside the ETF queue for a
while, then when the txtime is near, they pushed to the NIC's buffer. I
want to skip the ETF part to see how the NIC behave on its own. Its a
black box right now, but we want to see some edge case behavior, since
we will have mixed host and AF_PACKET/XDP traffic in our experiment. 

> 
> > When we bypassed ETF, everything seemed ok regarding the timing:
> > our
> > packet received about +/-15ns offset at the receiver (now its the
> > same
> > machine just to make sure with the timesync) compared to the
> > timestamp
> > set with SO_TXTIME CMSG.
> > 
> > What we tried now is to bypass the ETF qdisc. We enabled the ETF
> > qdisc
> > with hardware offload and sent the exact same packets, but this
> > time
> > with PACKET_QDISC_BAYPASS enabled on the AF_PACKET socket. The
> > codepath
> > looks good, the qdisc part is not called, the packet_snd calls the
> > dev_direct_xmit which calls the igc_xmit_frame. However, in this
> > case
> > the packet was sent more or less immediately.
> 
> Well, yeah the code path looks good indeed. packet_snd() copies the
> transmit time which is provided by the CMSG and calls into
> packet_xmit(), dev_direct_xmit()...
> 
> > 
> > I wonder why we do not see the delayed sending in this case? We
> > tried
> > with different offsets (e.g. 0.1, 0.01, 0.001 sec in the future)
> > but we
> > received the packet after 20-30usec every time. I cant see any code
> > that touches the skb timestamp after the packet_snd, so I suspect
> > that
> > the igc_xmit_frame sees the same timestamp that it would see in the
> > non-baypass case.
> 
> Maybe add some trace_printk()s to track what timestamps are actually
> calculated in igc_tx_launchtime() and if it makes sense?

Thanks, we looked into with the following script:

bpftrace -e 'kfunc:igc_xmit_frame{printf("%ld\n", args->skb->tstamp);}'

The timestamp looks correct for both cases. We will double check that,
but after a quick test the timestamp exactly what we set with SO_TXTIME
CMSG both qdisc and bypass cases.

> 
> Second point to make sure is that the Tx queue your packet is being
> transmitted to has Launch Time enabled.

Good point! We missed that. IIUC the TX queue selected by the skb-
>queue_mapping after the igc_xmit_frame part. We set that parameter by
skbedit tc  action but that is not executed in the bypass case. For
that reason, the skb->queue_mapping more or less random.

However, for a quick test, we set ETF offload for all 4 TX queues. So
no matter which queue the packet enqueued it should respect the
launchtime. But we dont really see this behavior... Nevertheless its a
good catch, I missed this aspect completely at the time of my first
email.

> 
> > 
> > I happen to have the i225 user manual, but after some grep I cannot
> > find any debug registers or counters to monitor the behavior of the
> > scheduled transmission (scheduling errors or bad timestamps, etc.).
> > Are
> > there any?
> 
> Not that I'm aware of.
> 
> > 
> > I am afraid this issue might also be relevant for the AF_XDP case,
> > which also hooks after the qdisc layer, so the launchtime (or
> > whatever
> > it is called) is handled directly by the igc driver.
> 
> Is that already possible with AF_XDP? There were some patches on
> xdp-hints, but i don't think it has been merged yet.

Not yet. There was a patchset with the infra and stmmac as an user,
right now I think its under some rework including igc as another user. 
However in principle, the codepath might be similar to AF_PACKET, the
only difference is the lunchtime metadata bundled with the xdp_buff and
not with the skb.

> 
> Thanks,
> Kurt

Best,
Ferenc

Re: igc: AF_PACKET and SO_TXTIME question

[Ferenc Fejes]

I think I'm figured out what was the issue, thanks for the help!

Solution inline.

On Thu, 2024-02-15 at 09:38 +0100, Fejes Ferenc wrote:
> 
> > 
> > Second point to make sure is that the Tx queue your packet is being
> > transmitted to has Launch Time enabled.
> 
> Good point! We missed that. IIUC the TX queue selected by the skb-
> > queue_mapping after the igc_xmit_frame part. We set that parameter
> > by
> skbedit tc  action but that is not executed in the bypass case. For
> that reason, the skb->queue_mapping more or less random.
> 
> However, for a quick test, we set ETF offload for all 4 TX queues. So
> no matter which queue the packet enqueued it should respect the
> launchtime. But we dont really see this behavior... Nevertheless its
> a
> good catch, I missed this aspect completely at the time of my first
> email.

...

We simply placed the ETF qdisc as the root qdisc and assumed that this
would enable offload on all TX queues. Apparently this is not the case.
According to the code, it was only enabled for queue 0. Another mistake
we made is we used multiq qdisc with skbedit queue_mapping without
bypass - which works since the tx action is executed.
However, with qdisc baypass, the TX queue selection for the packets
sent to the AF_PACKET socket looks like this

tx_queue = cpu_id of task % num_tx_queues.

With taskset, we were able to explicitly send packets to TX queue 0 in
the bypass case, and that essentially solved the problem.

So we switched to mqprio and enabled offloading on all queues and with
that we always see the delayed packet transmission with launchtime
enabled.

> > 
> > 

...

> > Is that already possible with AF_XDP? There were some patches on
> > xdp-hints, but i don't think it has been merged yet.
> 
> Not yet. There was a patchset with the infra and stmmac as an user,
> right now I think its under some rework including igc as another
> user. 
> However in principle, the codepath might be similar to AF_PACKET, the
> only difference is the lunchtime metadata bundled with the xdp_buff
> and
> not with the skb.

As a results the AF_XDP hints must be work as expected when its done.

> 
> > 
> > Thanks,
> > Kurt
> 
> Best,
> Ferenc
> 

Thanks,
Ferenc

Re: igc: AF_PACKET and SO_TXTIME question

[Kurt Kanzenbach]

[-- Attachment #1: Type: text/plain, Size: 921 bytes --]

Hi Ferenc,

On Fri Feb 16 2024, Ferenc Fejes wrote:
> We simply placed the ETF qdisc as the root qdisc and assumed that this
> would enable offload on all TX queues. Apparently this is not the case.
> According to the code, it was only enabled for queue 0. Another mistake
> we made is we used multiq qdisc with skbedit queue_mapping without
> bypass - which works since the tx action is executed.
> However, with qdisc baypass, the TX queue selection for the packets
> sent to the AF_PACKET socket looks like this
>
> tx_queue = cpu_id of task % num_tx_queues.
>
> With taskset, we were able to explicitly send packets to TX queue 0 in
> the bypass case, and that essentially solved the problem.
>
> So we switched to mqprio and enabled offloading on all queues and with
> that we always see the delayed packet transmission with launchtime
> enabled.

Glad it worked out. Thanks for sharing your solution!

Thanks,
Kurt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]