Wireguard head of line blocking when CPUs saturate

Wireguard head of line blocking when CPUs saturate

[Toke Høiland-Jørgensen]

Hey everyone

I'm running Wireguard on my main gateway, which is a not-super-high
powered ARM box with eight cores (based on the NXP LS1088A SoC). The box
does, however, also have eight hardware queues for its networking, which
means regular network traffic can be spread nicely across the cores.

However, the per-core performance is limited, making it pretty trivial
to saturate a single core by just running a fat TCP flow through it. And
when this happens, Wireguard traffic just... stalls. I.e., no traffic
gets through the Wireguard interface until the (unrelated) flow
saturating one of the cores subsides.

I suspect what happens is that Wireguard spreads out traffic to all
cores for encryption, but has to wait for the respective CPUs to finish
encrypting the packets in order before they can actually be transmitted.
And because one CPU is now suddenly saturated in softirq context, the
Wireguard work queue never gets a chance to run on that CPU, stalling TX
progress for the Wireguard device entirely.

I'm sending this message to (a) see if anyone else is seeing the same
kind of stalling, and (b) to get input on whether the explanation
outlined above seems plausible. And, in the case of affirmative answers
to both (a) and (b), to hopefully start a discussion on what to do about
this :)

-Toke

Re: Wireguard head of line blocking when CPUs saturate

[Markus Reichelt]

* Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> I'm sending this message to (a) see if anyone else is seeing the same
> kind of stalling, and (b) to get input on whether the explanation
> outlined above seems plausible. And, in the case of affirmative answers
> to both (a) and (b), to hopefully start a discussion on what to do about
> this :)

I don't use a wireguard setup on arm hardware, I am just brainstorming...
Given that you just use one tunnel, how about pinning wireguard to a core
and, more importantly, pin the program that causes the 'fat tcp flow' to
some other core?

What you describe must be a common problem on low performance ARM setups
running wg on openWRT (mikrotik comes to mind), so I'd have a look in that
corner of the internet also ;)

HTH