Re: [PATCH net-next 1/3] net: provide macros for commonly copied lockless queue stop/wake code

[Heiner Kallweit <hkallweit1@gmail.com>]

On 01.04.2023 20:58, Jakub Kicinski wrote:
> On Sat, 1 Apr 2023 17:18:12 +0200 Heiner Kallweit wrote:
>>> +#define __netif_tx_queue_maybe_wake(txq, get_desc, start_thrs, down_cond) \
>>> +	({								\
>>> +		int _res;						\
>>> +									\
>>> +		_res = -1;						\  
>>
>> One more question: Don't we need a read memory barrier here to ensure
>> get_desc is up-to-date?
> 
> CC: Alex, maybe I should not be posting after 10pm, with the missing v2
> and sparse CC list.. :|
> 
> I was thinking about this too yesterday. AFAICT this implementation
> could indeed result in waking even tho the queue is full on non-x86.
> That's why the drivers have an extra check at the start of .xmit? :(
> 
> I *think* that the right ordering would be:
> 
> WRITE cons
> mb()  # A
> READ stopped
> rmb() # C
> READ prod, cons
> 
> And on the producer side (existing):
> 
> WRITE prod
> READ prod, cons
> mb()  # B
> WRITE stopped
> READ prod, cons
> 
> But I'm slightly afraid to change it, it's been working for over 
> a decade :D
> 
> One neat thing that I noticed, which we could potentially exploit 
> if we were to touch this code is that BQL already has a smp_mb() 
> on the consumer side. So on any kernel config and driver which support
> BQL we can use that instead of adding another barrier at #A.
> 
To me it seems ixgbe relies on this BQL-related barrier in
netdev_tx_completed_queue, maybe unintentionally.

I wonder whether we should move the smp_mb() from __netif_tx_queue_try_wake
to __netif_tx_queue_maybe_wake, before accessing get_desc.
Answer may depend on whether there's any use case where drivers would
call the "try" version directly, instead of the "may" version.
And, just thinking out loud, maybe add a flag to the macro to disable
the smp_mb() call, for cases where the driver has the required barrier
already. Setting this flag may look like this.

bool disable_barrier_in_may_wake = IS_ENABLED(CONFIG_BQL);

However this is quite some effort just to avoid a duplicated barrier.

> It would actually be a neat optimization because right now, AFAICT,
> completion will fire the # A -like barrier almost every time.
> 
>>> +		if (likely(get_desc > start_thrs))			\
>>> +			_res = __netif_tx_queue_try_wake(txq, get_desc,	\
>>> +							 start_thrs,	\
>>> +							 down_cond);	\
>>> +		_res;							\
>>> +	})