* pull-request: ieee802154 2017-10-18
From: Stefan Schmidt @ 2017-10-18 22:03 UTC (permalink / raw)
To: davem; +Cc: linux-wpan, alex.aring, marcel, netdev, Stefan Schmidt
From: Stefan Schmidt <stefan@osg.samsung.com>
Hello Dave.
Please find below a pull request from the ieee802154 subsystem for net-next.
regards
Stefan Schmidt
The following changes since commit b9f1f1ce866c28e3d9b86202441b220244754a69:
tcp: fix tcp_xmit_retransmit_queue() after rbtree introduction (2017-10-18 14:19:26 +0100)
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/sschmidt/wpan-next.git ieee802154-for-davem-2017-10-18
for you to fetch changes up to 396665e8320987ff43b20a62a6a1cdae57aa1cc1:
Merge remote-tracking branch 'net-next/master' (2017-10-18 17:40:18 +0200)
----------------------------------------------------------------
A warning fix from Colin King and a typo fix from Xue Liu. Bundled together with
a fixed firmware version check in the atusb driver.
----------------------------------------------------------------
Colin Ian King (1):
ieee802154: atusb: make two structures static, fixes warnings
Stefan Schmidt (2):
ieee802154: atusb: fix firmware version check to enable frame retries
Merge remote-tracking branch 'net-next/master'
Xue Liu (1):
ieee802154: netlink: fix typo of the name of struct genl_ops
drivers/net/ieee802154/atusb.c | 8 +++-----
net/ieee802154/netlink.c | 6 +++---
2 files changed, 6 insertions(+), 8 deletions(-)
^ permalink raw reply
* [PATCH net] packet: avoid panic in packet_getsockopt()
From: Eric Dumazet @ 2017-10-18 22:08 UTC (permalink / raw)
To: David Miller; +Cc: netdev, Willem de Bruijn, Eric Dumazet
From: Eric Dumazet <edumazet@google.com>
syzkaller got crashes in packet_getsockopt() processing
PACKET_ROLLOVER_STATS command while another thread was managing
to change po->rollover
Using RCU will fix this bug. We might later add proper RCU annotations
for sparse sake.
Fixes: a9b6391814d5 ("packet: rollover statistics")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Willem de Bruijn <willemb@google.com>
---
net/packet/af_packet.c | 22 +++++++++++++++-------
1 file changed, 15 insertions(+), 7 deletions(-)
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index bec01a3daf5b02bd716dbff5c9efef8d6a7982be..1d8a7add86b4f29880e11c6f4971d79319dcb426 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -1796,8 +1796,10 @@ static struct packet_fanout *fanout_release(struct sock *sk)
else
f = NULL;
- if (po->rollover)
+ if (po->rollover) {
kfree_rcu(po->rollover, rcu);
+ po->rollover = NULL;
+ }
}
mutex_unlock(&fanout_mutex);
@@ -3851,6 +3853,7 @@ static int packet_getsockopt(struct socket *sock, int level, int optname,
void *data = &val;
union tpacket_stats_u st;
struct tpacket_rollover_stats rstats;
+ struct packet_rollover *rollover;
if (level != SOL_PACKET)
return -ENOPROTOOPT;
@@ -3929,13 +3932,18 @@ static int packet_getsockopt(struct socket *sock, int level, int optname,
0);
break;
case PACKET_ROLLOVER_STATS:
- if (!po->rollover)
+ rcu_read_lock();
+ rollover = rcu_dereference(po->rollover);
+ if (rollover) {
+ rstats.tp_all = atomic_long_read(&rollover->num);
+ rstats.tp_huge = atomic_long_read(&rollover->num_huge);
+ rstats.tp_failed = atomic_long_read(&rollover->num_failed);
+ data = &rstats;
+ lv = sizeof(rstats);
+ }
+ rcu_read_unlock();
+ if (!rollover)
return -EINVAL;
- rstats.tp_all = atomic_long_read(&po->rollover->num);
- rstats.tp_huge = atomic_long_read(&po->rollover->num_huge);
- rstats.tp_failed = atomic_long_read(&po->rollover->num_failed);
- data = &rstats;
- lv = sizeof(rstats);
break;
case PACKET_TX_HAS_OFF:
val = po->tp_tx_has_off;
^ permalink raw reply related
* Re: [PATCH RFC V1 net-next 0/6] Time based packet transmission
From: Jesus Sanchez-Palencia @ 2017-10-18 22:18 UTC (permalink / raw)
To: Richard Cochran
Cc: netdev, linux-kernel, intel-wired-lan, Andre Guedes,
Anna-Maria Gleixner, David Miller, Henrik Austad, John Stultz,
Thomas Gleixner, Vinicius Costa Gomes, Briano, Ivan, Levi Pearson
In-Reply-To: <cover.1505719061.git.rcochran@linutronix.de>
Hi Richard,
On 09/18/2017 12:41 AM, Richard Cochran wrote:
> This series is an early RFC that introduces a new socket option
> allowing time based transmission of packets. This option will be
> useful in implementing various real time protocols over Ethernet,
> including but not limited to P802.1Qbv, which is currently finding
> its way into 802.1Q.
>
> * Open questions about SO_TXTIME semantics
>
> - What should the kernel do if the dialed Tx time is in the past?
> Should the packet be sent ASAP, or should we throw an error?
>
> - Should the kernel inform the user if it detects a missed deadline,
> via the error queue for example?
>
> - What should the timescale be for the dialed Tx time? Should the
> kernel select UTC when using the SW Qdisc and the HW time
> otherwise? Or should the socket option include a clockid_t?
>
> * Things todo
>
> - Design a Qdisc for purpose of configuring SO_TXTIME. There should
> be one option to dial HW offloading or SW best effort.
>
> - Implement the SW best effort variant. Here is my back of the
> napkin sketch. Each interface has its own timerqueue keeping the
> TXTIME packets in order and a FIFO for all other traffic. A guard
> window starts at the earliest deadline minus the maximum MTU minus
> a configurable fudge factor. The Qdisc uses a hrtimer to transmit
> the next packet in the timerqueue. During the guard window, all
> other traffic is defered unless the next packet can be transmitted
> before the guard window expires.
Even for HW offloading this timerqueue could be used for enforcing that packets
are always sorted by their launch time when they get enqueued into the
netdevice. Of course, assuming that this would be something that we'd like to
provide from within the kernel.
>
> * Current limitations
>
> - The driver does not handle out of order packets. If user space
> sends a packet with an earlier Tx time, then the code should stop
> the queue, reshuffle the descriptors accordingly, and then
> restart the queue.
Wouldn't be an issue if the above was provided.
>
> - The driver does not correctly queue up packets in the distant
> future. The i210 has a limited time window of +/- 0.5 seconds.
> Packets with a Tx time greater than that should be deferred in
> order to enqueue them later on.
>
> * Performance measurements
>
> 1. Prepared a PC and the Device Under Test (DUT) each with an Intel
> i210 card connected with a crossover cable.
> 2. The DUT was a Pentium(R) D CPU 2.80GHz running PREEMPT_RT
> 4.9.40-rt30 with about 50 usec maximum latency under cyclictest.
> 3. Synchronized the DUT's PHC to the PC's PHC using ptp4l.
> 4. Synchronized the DUT's system clock to its PHC using phc2sys.
> 5. Started netperf to produce some network load.
> 6. Measured the arrival time of the packets at the PC's PHC using
> hardware time stamping.
>
> I ran ten minute tests both with and without using the so_txtime
> option, with a period was 1 millisecond. I then repeated the
> so_txtime case but with a 250 microsecond period. The measured
> offset from the expected period (in nanoseconds) is shown in the
> following table.
>
> | | plain preempt_rt | so_txtime | txtime @ 250 us |
> |---------+------------------+---------------+-----------------|
> | min: | +1.940800e+04 | +4.720000e+02 | +4.720000e+02 |
> | max: | +7.556000e+04 | +5.680000e+02 | +5.760000e+02 |
> | pk-pk: | +5.615200e+04 | +9.600000e+01 | +1.040000e+02 |
> | mean: | +3.292776e+04 | +5.072274e+02 | +5.073602e+02 |
> | stddev: | +6.514709e+03 | +1.310849e+01 | +1.507144e+01 |
> | count: | 600000 | 600000 | 2400000 |
>
> Using so_txtime, the peak to peak jitter is about 100 nanoseconds,
> independent of the period. In contrast, plain preempt_rt shows a
> jitter of of 56 microseconds. The average delay of 507 nanoseconds
> when using so_txtime is explained by the documented input and output
> delays on the i210 cards.
This is great. Just out of curiosity, were you using vlans on your tests?
>
> The test program is appended, below. If anyone is interested in
> reproducing this test, I can provide helper scripts.
I might try to reproduce them soon. I would appreciate if you could provide me
with the scripts, please.
Thanks,
Jesus
>
> Thanks,
> Richard
>
>
> Richard Cochran (6):
> net: Add a new socket option for a future transmit time.
> net: skbuff: Add a field to support time based transmission.
> net: ipv4: raw: Hook into time based transmission.
> net: ipv4: udp: Hook into time based transmission.
> net: packet: Hook into time based transmission.
> net: igb: Implement time based transmission.
>
> arch/alpha/include/uapi/asm/socket.h | 3 ++
> arch/frv/include/uapi/asm/socket.h | 3 ++
> arch/ia64/include/uapi/asm/socket.h | 3 ++
> arch/m32r/include/uapi/asm/socket.h | 3 ++
> arch/mips/include/uapi/asm/socket.h | 3 ++
> arch/mn10300/include/uapi/asm/socket.h | 3 ++
> arch/parisc/include/uapi/asm/socket.h | 3 ++
> arch/powerpc/include/uapi/asm/socket.h | 3 ++
> arch/s390/include/uapi/asm/socket.h | 3 ++
> arch/sparc/include/uapi/asm/socket.h | 3 ++
> arch/xtensa/include/uapi/asm/socket.h | 3 ++
> drivers/net/ethernet/intel/igb/e1000_82575.h | 1 +
> drivers/net/ethernet/intel/igb/e1000_defines.h | 68 +++++++++++++++++++++++++-
> drivers/net/ethernet/intel/igb/e1000_regs.h | 5 ++
> drivers/net/ethernet/intel/igb/igb.h | 3 +-
> drivers/net/ethernet/intel/igb/igb_main.c | 68 +++++++++++++++++++++++---
> include/linux/skbuff.h | 2 +
> include/net/sock.h | 2 +
> include/uapi/asm-generic/socket.h | 3 ++
> net/core/sock.c | 12 +++++
> net/ipv4/raw.c | 2 +
> net/ipv4/udp.c | 5 +-
> net/packet/af_packet.c | 6 +++
> 23 files changed, 200 insertions(+), 10 deletions(-)
>
^ permalink raw reply
* Re: Linux 4.12+ memory leak on router with i40e NICs
From: Paweł Staszewski @ 2017-10-18 22:20 UTC (permalink / raw)
To: Alexander Duyck
Cc: Pavlos Parissis, Anders K. Pedersen | Cohaesio,
netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org,
alexander.h.duyck@intel.com
In-Reply-To: <748b6d9d-4a3f-4eaa-ad24-27060c2e2642@itcare.pl>
W dniu 2017-10-18 o 17:44, Paweł Staszewski pisze:
>
>
> W dniu 2017-10-17 o 16:08, Paweł Staszewski pisze:
>>
>>
>> W dniu 2017-10-17 o 13:52, Paweł Staszewski pisze:
>>>
>>>
>>> W dniu 2017-10-17 o 13:05, Paweł Staszewski pisze:
>>>>
>>>>
>>>> W dniu 2017-10-17 o 12:59, Paweł Staszewski pisze:
>>>>>
>>>>>
>>>>> W dniu 2017-10-17 o 12:51, Paweł Staszewski pisze:
>>>>>>
>>>>>>
>>>>>> W dniu 2017-10-17 o 12:20, Paweł Staszewski pisze:
>>>>>>>
>>>>>>>
>>>>>>> W dniu 2017-10-17 o 11:48, Paweł Staszewski pisze:
>>>>>>>>
>>>>>>>>
>>>>>>>> W dniu 2017-10-17 o 02:44, Paweł Staszewski pisze:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> W dniu 2017-10-17 o 01:56, Alexander Duyck pisze:
>>>>>>>>>> On Mon, Oct 16, 2017 at 4:34 PM, Paweł Staszewski
>>>>>>>>>> <pstaszewski@itcare.pl> wrote:
>>>>>>>>>>>
>>>>>>>>>>> W dniu 2017-10-16 o 18:26, Paweł Staszewski pisze:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> W dniu 2017-10-16 o 13:20, Pavlos Parissis pisze:
>>>>>>>>>>>>> On 15/10/2017 02:58 πμ, Alexander Duyck wrote:
>>>>>>>>>>>>>> Hi Pawel,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To clarify is that Dave Miller's tree or Linus's that you
>>>>>>>>>>>>>> are talking
>>>>>>>>>>>>>> about? If it is Dave's tree how long ago was it you
>>>>>>>>>>>>>> pulled it since I
>>>>>>>>>>>>>> think the fix was just pushed by Jeff Kirsher a few days
>>>>>>>>>>>>>> ago.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The issue should be fixed in the following commit:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/drivers/net/ethernet/intel/i40e/i40e_txrx.c?id=2b9478ffc550f17c6cd8c69057234e91150f5972
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Do you know when it is going to be available on net-next
>>>>>>>>>>>>> and linux-stable
>>>>>>>>>>>>> repos?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Pavlos
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> I will make some tests today night with "net" git tree
>>>>>>>>>>>> where this patch is
>>>>>>>>>>>> included.
>>>>>>>>>>>> Starting from 0:00 CET
>>>>>>>>>>>> :)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Upgraded and looks like problem is not solved with that patch
>>>>>>>>>>> Currently running system with
>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/
>>>>>>>>>>> kernel
>>>>>>>>>>>
>>>>>>>>>>> Still about 0.5GB of memory is leaking somewhere
>>>>>>>>>>>
>>>>>>>>>>> Also can confirm that the latest kernel where memory is not
>>>>>>>>>>> leaking (with
>>>>>>>>>>> use i40e driver intel 710 cards) is 4.11.12
>>>>>>>>>>> With kernel 4.11.12 - after hour no change in memory usage.
>>>>>>>>>>>
>>>>>>>>>>> also checked that with ixgbe instead of i40e with same
>>>>>>>>>>> net.git kernel there
>>>>>>>>>>> is no memleak - after hour same memory usage - so for 100%
>>>>>>>>>>> this is i40e
>>>>>>>>>>> driver problem.
>>>>>>>>>> So how long was the run to get the .5GB of memory leaking?
>>>>>>>>> 1 hour
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Also is there any chance of you being able to bisect to
>>>>>>>>>> determine
>>>>>>>>>> where the memory leak was introduced since as you pointed out it
>>>>>>>>>> didn't exist in 4.11.12 so odds are it was introduced somewhere
>>>>>>>>>> between 4.11 and the latest kernel release.
>>>>>>>>> Can be hard cause currently need to back to 4.11.12 - this is
>>>>>>>>> production host/router
>>>>>>>>> Will try to find some free/test router for tests/bicects with
>>>>>>>>> i40e driver (intel 710 cards)
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> - Alex
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Also forgoto to add errors for i40e when driver initialize:
>>>>>>>> [ 15.760569] i40e 0000:02:00.1: Error I40E_AQ_RC_ENOSPC
>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>> [ 16.365587] i40e 0000:03:00.3: Error I40E_AQ_RC_ENOSPC
>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>> [ 16.367686] i40e 0000:02:00.2: Error I40E_AQ_RC_ENOSPC
>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>> [ 16.368816] i40e 0000:03:00.0: Error I40E_AQ_RC_ENOSPC
>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>> [ 16.369877] i40e 0000:03:00.2: Error I40E_AQ_RC_ENOSPC
>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>> [ 16.370941] i40e 0000:02:00.3: Error I40E_AQ_RC_ENOSPC
>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>> [ 16.372005] i40e 0000:02:00.0: Error I40E_AQ_RC_ENOSPC
>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>> [ 16.373029] i40e 0000:03:00.1: Error I40E_AQ_RC_ENOSPC
>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>
>>>>>>>> some params that are set for this nic's
>>>>>>>> ip link set up dev $i
>>>>>>>> ethtool -A $i autoneg off rx off tx off
>>>>>>>> ethtool -G $i rx 1024 tx 2048
>>>>>>>> ip link set $i txqueuelen 1000
>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>>> 512 tx-usecs 128
>>>>>>>> ethtool -L $i combined 6
>>>>>>>> #ethtool -N $i rx-flow-hash udp4 sdfn
>>>>>>>> ethtool -K $i ntuple on
>>>>>>>> ethtool -K $i gro off
>>>>>>>> ethtool -K $i tso off
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Also after TSO/GRO on there is memory usage change - and leaking
>>>>>>> faster
>>>>>>> Below image from memory usage before change with TSO/GRO OFF and
>>>>>>> after enabling TSO/GRO
>>>>>>>
>>>>>>> https://ibb.co/dTqBY6
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>> Pawel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> With settings like this:
>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>> enp3s0f2 enp3s0f3'
>>>>>> for i in $ifc
>>>>>> do
>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>> 512 tx-usecs 128
>>>>>> ethtool -K $i gro on
>>>>>> ethtool -K $i tso on
>>>>>>
>>>>>> done
>>>>>>
>>>>>> Server is leaking about 4-6MB per each 10 seconds
>>>>>> MEMLEAK:
>>>>>> 5 MB/10sec
>>>>>> 6 MB/10sec
>>>>>> 4 MB/10sec
>>>>>> 4 MB/10sec
>>>>>>
>>>>>>
>>>>>> Other settings TSO/GRO off
>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>> enp3s0f2 enp3s0f3'
>>>>>> for i in $ifc
>>>>>> do
>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>> 512 tx-usecs 128
>>>>>> ethtool -K $i gro off
>>>>>> ethtool -K $i tso off
>>>>>>
>>>>>> done
>>>>>>
>>>>>> Same leak about 5MB per 10 seconds
>>>>>> MEMLEAK:
>>>>>> 5 MB/10sec
>>>>>> 5 MB/10sec
>>>>>> 5 MB/10sec
>>>>>>
>>>>>>
>>>>>> Other settings rx-usecs change from 512 to 1024:
>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>> enp3s0f2 enp3s0f3'
>>>>>> for i in $ifc
>>>>>> do
>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>> 1024 tx-usecs 128
>>>>>> ethtool -K $i gro off
>>>>>> ethtool -K $i tso off
>>>>>>
>>>>>> done
>>>>>>
>>>>>> MEMLEAK:
>>>>>> 4 MB/10sec
>>>>>> 3 MB/10sec
>>>>>> 4 MB/10sec
>>>>>> 4 MB/10sec
>>>>>>
>>>>>>
>>>>>> So memleak have something to do with rx-usecs (less interrupts
>>>>>> but bigger latency for traffic)
>>>>>>
>>>>>>
>>>>>> But also enabling TSO/GRO making leak about 1MB bigger for each
>>>>>> 10 seconds
>>>>>>
>>>>>>
>>>>>>
>>>>> So far best config is:
>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>> enp3s0f2 enp3s0f3'
>>>>> for i in $ifc
>>>>> do
>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 64
>>>>> tx-usecs 512
>>>>> ethtool -K $i gro off
>>>>> ethtool -K $i tso on
>>>>>
>>>>> done
>>>>>
>>>>> MEMLEAK - about 2MB/10secs
>>>>> 2 MB/10sec
>>>>> 2 MB/10sec
>>>>> 2 MB/10sec
>>>>>
>>>>>
>>>>> With - rx-usecs set to 256 (about 7-9MB/10secs memleak)
>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>> enp3s0f2 enp3s0f3'
>>>>> for i in $ifc
>>>>> do
>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 256
>>>>> tx-usecs 512
>>>>> ethtool -K $i gro off
>>>>> ethtool -K $i tso on
>>>>>
>>>>> done
>>>>>
>>>>> MEMLEAK:
>>>>> 7 MB/10sec
>>>>> 7 MB/10sec
>>>>> 8 MB/10sec
>>>>> 9 MB/10sec
>>>>>
>>>>>
>>>>
>>>> And even less memleak with rx-usecs set to 32
>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>> enp3s0f3'
>>>> for i in $ifc
>>>> do
>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 32
>>>> tx-usecs 512
>>>> ethtool -K $i gro off
>>>> ethtool -K $i tso on
>>>>
>>>> done
>>>>
>>>>
>>>> MEMLEAK - about 0-2MB for each 10 seconds
>>>> 0 MB/10sec
>>>> 1 MB/10sec
>>>> 0 MB/10sec
>>>> 2 MB/10sec
>>>> 1 MB/10sec
>>>>
>>>>
>>>>
>>>
>>>
>>> So best settings - to have as less leak as possible for now
>>> (rx-usecs set to 16):
>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>> enp3s0f3'
>>> for i in $ifc
>>> do
>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 16
>>> tx-usecs 768
>>> ethtool -K $i gro on
>>> ethtool -K $i tso on
>>>
>>> done
>>>
>>>
>>> MEMLEAK: (0-1MB/10seconds)
>>> 0 MB/10sec
>>> 0 MB/10sec
>>> 0 MB/10sec
>>> 1 MB/10sec
>>> 1 MB/10sec
>>> -1 MB/10sec
>>> 1 MB/10sec
>>> 1 MB/10sec
>>> 0 MB/10sec
>>>
>>> (there are some memory recycles - so this is good :) )
>>>
>>>
>>>
>>> Compared to(rx-usecs 512):
>>>
>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>> enp3s0f3'
>>> for i in $ifc
>>> do
>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 512
>>> tx-usecs 128
>>> ethtool -K $i gro on
>>> ethtool -K $i tso on
>>>
>>> done
>>>
>>> Server is leaking about 4-6MB per each 10 seconds
>>> MEMLEAK:
>>> 5 MB/10sec
>>> 6 MB/10sec
>>> 4 MB/10sec
>>> 4 MB/10sec
>>>
>>>
>>
>> And graph where all changes for rx-usecs was done over some time:
>> https://ibb.co/nrRfbR
>>
>>
>>
>>
>>
> Cant eliminate the problem with settings - memleak is bigger or less
> visible with rx-usecs set to low values - but then have 100% cpu load
> - cant have rx-usecs set to 16
>
> Cant find also other host with same cards or that are using i40e
> driver for tests with bisecting
> So will just replace to mellanox :)
>
>
Also after fresh reboot with i40e
startup settings:
ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
enp3s0f3'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 2048 tx 2048
ip link set $i txqueuelen 1000
#ethtool -C $i rx-usecs 256
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17
tx-usecs 125
ethtool -L $i combined 6
#ethtool -N $i rx-flow-hash udp4 sdfn
#ethtool -K $i ntuple on
#ethtool -K $i gro off
#ethtool -K $i tso off
done
After issuing:
ethtool -K enp2s0f0 gro on tso on
dmesg shows
[35764.338259] i40e 0000:02:00.0: PF reset failed, -15
and no traffic on the card :)
^ permalink raw reply
* Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
From: Jesus Sanchez-Palencia @ 2017-10-18 22:37 UTC (permalink / raw)
To: Richard Cochran
Cc: Vinicius Costa Gomes, netdev, jhs, xiyou.wangcong, jiri,
intel-wired-lan, andre.guedes, ivan.briano, boon.leong.ong,
Levi Pearson, Henrik Austad
In-Reply-To: <20170920052558.h6c4lkqmzk2h2pdq@localhost>
Hi Richard,
On 09/19/2017 10:25 PM, Richard Cochran wrote:
(...)
>
>> I have a question, what about a controller that doesn't provide a way to
>> set a per-packet transmission time, but it supports Qbv/Qbu. What would
>> be your proposal to configure it?
>
> SO_TXTIME will have a generic SW fallback.
>
> BTW, regarding the i210, there is no sensible way to configure both
> CBS and time based transmission at the same time. The card performs a
> logical AND to make the launch decision. The effect of this is that
> each and every packet needs a LaunchTime, and the driver would be
> forced to guess the time for a packet before entering it into its
> queue.
>
> So if we end up merging CBS and SO_TXTIME, then we'll have to make
> them exclusive of each other (in the case of the i210) and manage the
> i210 queue configurations correctly.
>
I've ran some quick tests here having launch time enabled on i210 + our cbs
patchset. When valid Launch times are set on each packet you still get the
expected behavior, so I'm not sure we should just make them exclusive of each other.
I also did some tests with when you don't set valid launch times, but here using
your idea from above, so with the driver calculating a valid launch time (i.e.
current NIC time + X ns, varying X across tests) for packets that didn't have it
set by the user, and I wasn't too happy with its reliability. It could
definitely be improved, but it has left me wondering: instead, what about
documenting that if you enable TXTIME, then you *must* provide a valid Launch
time for all packets on traffic classes that are affected?
With the SO_TXTIME qdisc idea in place, that could even be enforced before
packets were enqueued into the netdevice.
Regards,
Jesus
^ permalink raw reply
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
From: Paweł Staszewski @ 2017-10-18 22:45 UTC (permalink / raw)
To: Eric Dumazet
Cc: Florian Fainelli, Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
In-Reply-To: <1508363646.31614.145.camel@edumazet-glaptop3.roam.corp.google.com>
W dniu 2017-10-18 o 23:54, Eric Dumazet pisze:
> On Wed, 2017-10-18 at 23:49 +0200, Paweł Staszewski wrote:
>
>> How far it is from applying this to the kernel ?
>>
>> So far im using this on all my servers from about 3 months now without
>> problems
> It is a hack, and does not support properly bonding/team.
>
> ( If the real_dev->privflags IFF_XMIT_DST_RELEASE bit changes,
> we want to update all the vlans at the same time )
>
> We need something more sophisticated, and I had no time to spend on
> this topic recently.
>
>
>
>
>
ok
^ permalink raw reply
* [next-queue PATCH] net/sched/sch_cbs: Fix compilation on 32bit architectures
From: Vinicius Costa Gomes @ 2017-10-18 22:47 UTC (permalink / raw)
To: netdev, intel-wired-lan
Cc: Vinicius Costa Gomes, jeffrey.t.kirsher, jesus.sanchez-palencia
There was still a couple of divisions of 64bit quantities happening,
which can fail to compile if there aren't instructions to handle that
kind of division.
It will fail with a message like this:
ERROR: "__aeabi_ldivmod" [net/sched/sch_cbs.ko] undefined!
Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
---
net/sched/sch_cbs.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/net/sched/sch_cbs.c b/net/sched/sch_cbs.c
index cae021c642e5..bdb533b7fb8c 100644
--- a/net/sched/sch_cbs.c
+++ b/net/sched/sch_cbs.c
@@ -331,8 +331,8 @@ static int cbs_dump(struct Qdisc *sch, struct sk_buff *skb)
opt.hicredit = q->hicredit;
opt.locredit = q->locredit;
- opt.sendslope = q->sendslope / BYTES_PER_KBIT;
- opt.idleslope = q->idleslope / BYTES_PER_KBIT;
+ opt.sendslope = div64_s64(q->sendslope, BYTES_PER_KBIT);
+ opt.idleslope = div64_s64(q->idleslope, BYTES_PER_KBIT);
opt.offload = q->offload;
if (nla_put(skb, TCA_CBS_PARMS, sizeof(opt), &opt))
--
2.14.2
^ permalink raw reply related
* Re: Linux 4.12+ memory leak on router with i40e NICs
From: Paweł Staszewski @ 2017-10-18 22:50 UTC (permalink / raw)
To: Alexander Duyck
Cc: Pavlos Parissis, Anders K. Pedersen | Cohaesio,
netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org,
alexander.h.duyck@intel.com
In-Reply-To: <16c9fa34-252f-e5a3-8b15-e5a8c4d8a46f@itcare.pl>
W dniu 2017-10-19 o 00:20, Paweł Staszewski pisze:
>
>
> W dniu 2017-10-18 o 17:44, Paweł Staszewski pisze:
>>
>>
>> W dniu 2017-10-17 o 16:08, Paweł Staszewski pisze:
>>>
>>>
>>> W dniu 2017-10-17 o 13:52, Paweł Staszewski pisze:
>>>>
>>>>
>>>> W dniu 2017-10-17 o 13:05, Paweł Staszewski pisze:
>>>>>
>>>>>
>>>>> W dniu 2017-10-17 o 12:59, Paweł Staszewski pisze:
>>>>>>
>>>>>>
>>>>>> W dniu 2017-10-17 o 12:51, Paweł Staszewski pisze:
>>>>>>>
>>>>>>>
>>>>>>> W dniu 2017-10-17 o 12:20, Paweł Staszewski pisze:
>>>>>>>>
>>>>>>>>
>>>>>>>> W dniu 2017-10-17 o 11:48, Paweł Staszewski pisze:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> W dniu 2017-10-17 o 02:44, Paweł Staszewski pisze:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> W dniu 2017-10-17 o 01:56, Alexander Duyck pisze:
>>>>>>>>>>> On Mon, Oct 16, 2017 at 4:34 PM, Paweł Staszewski
>>>>>>>>>>> <pstaszewski@itcare.pl> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> W dniu 2017-10-16 o 18:26, Paweł Staszewski pisze:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> W dniu 2017-10-16 o 13:20, Pavlos Parissis pisze:
>>>>>>>>>>>>>> On 15/10/2017 02:58 πμ, Alexander Duyck wrote:
>>>>>>>>>>>>>>> Hi Pawel,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To clarify is that Dave Miller's tree or Linus's that
>>>>>>>>>>>>>>> you are talking
>>>>>>>>>>>>>>> about? If it is Dave's tree how long ago was it you
>>>>>>>>>>>>>>> pulled it since I
>>>>>>>>>>>>>>> think the fix was just pushed by Jeff Kirsher a few days
>>>>>>>>>>>>>>> ago.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The issue should be fixed in the following commit:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/drivers/net/ethernet/intel/i40e/i40e_txrx.c?id=2b9478ffc550f17c6cd8c69057234e91150f5972
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do you know when it is going to be available on net-next
>>>>>>>>>>>>>> and linux-stable
>>>>>>>>>>>>>> repos?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Pavlos
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> I will make some tests today night with "net" git tree
>>>>>>>>>>>>> where this patch is
>>>>>>>>>>>>> included.
>>>>>>>>>>>>> Starting from 0:00 CET
>>>>>>>>>>>>> :)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Upgraded and looks like problem is not solved with that patch
>>>>>>>>>>>> Currently running system with
>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/
>>>>>>>>>>>> kernel
>>>>>>>>>>>>
>>>>>>>>>>>> Still about 0.5GB of memory is leaking somewhere
>>>>>>>>>>>>
>>>>>>>>>>>> Also can confirm that the latest kernel where memory is not
>>>>>>>>>>>> leaking (with
>>>>>>>>>>>> use i40e driver intel 710 cards) is 4.11.12
>>>>>>>>>>>> With kernel 4.11.12 - after hour no change in memory usage.
>>>>>>>>>>>>
>>>>>>>>>>>> also checked that with ixgbe instead of i40e with same
>>>>>>>>>>>> net.git kernel there
>>>>>>>>>>>> is no memleak - after hour same memory usage - so for 100%
>>>>>>>>>>>> this is i40e
>>>>>>>>>>>> driver problem.
>>>>>>>>>>> So how long was the run to get the .5GB of memory leaking?
>>>>>>>>>> 1 hour
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Also is there any chance of you being able to bisect to
>>>>>>>>>>> determine
>>>>>>>>>>> where the memory leak was introduced since as you pointed
>>>>>>>>>>> out it
>>>>>>>>>>> didn't exist in 4.11.12 so odds are it was introduced somewhere
>>>>>>>>>>> between 4.11 and the latest kernel release.
>>>>>>>>>> Can be hard cause currently need to back to 4.11.12 - this is
>>>>>>>>>> production host/router
>>>>>>>>>> Will try to find some free/test router for tests/bicects with
>>>>>>>>>> i40e driver (intel 710 cards)
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> - Alex
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Also forgoto to add errors for i40e when driver initialize:
>>>>>>>>> [ 15.760569] i40e 0000:02:00.1: Error I40E_AQ_RC_ENOSPC
>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>> [ 16.365587] i40e 0000:03:00.3: Error I40E_AQ_RC_ENOSPC
>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>> [ 16.367686] i40e 0000:02:00.2: Error I40E_AQ_RC_ENOSPC
>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>> [ 16.368816] i40e 0000:03:00.0: Error I40E_AQ_RC_ENOSPC
>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>> [ 16.369877] i40e 0000:03:00.2: Error I40E_AQ_RC_ENOSPC
>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>> [ 16.370941] i40e 0000:02:00.3: Error I40E_AQ_RC_ENOSPC
>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>> [ 16.372005] i40e 0000:02:00.0: Error I40E_AQ_RC_ENOSPC
>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>> [ 16.373029] i40e 0000:03:00.1: Error I40E_AQ_RC_ENOSPC
>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>
>>>>>>>>> some params that are set for this nic's
>>>>>>>>> ip link set up dev $i
>>>>>>>>> ethtool -A $i autoneg off rx off tx off
>>>>>>>>> ethtool -G $i rx 1024 tx 2048
>>>>>>>>> ip link set $i txqueuelen 1000
>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>>>> 512 tx-usecs 128
>>>>>>>>> ethtool -L $i combined 6
>>>>>>>>> #ethtool -N $i rx-flow-hash udp4 sdfn
>>>>>>>>> ethtool -K $i ntuple on
>>>>>>>>> ethtool -K $i gro off
>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Also after TSO/GRO on there is memory usage change - and
>>>>>>>> leaking faster
>>>>>>>> Below image from memory usage before change with TSO/GRO OFF
>>>>>>>> and after enabling TSO/GRO
>>>>>>>>
>>>>>>>> https://ibb.co/dTqBY6
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Pawel
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> With settings like this:
>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>> for i in $ifc
>>>>>>> do
>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>> 512 tx-usecs 128
>>>>>>> ethtool -K $i gro on
>>>>>>> ethtool -K $i tso on
>>>>>>>
>>>>>>> done
>>>>>>>
>>>>>>> Server is leaking about 4-6MB per each 10 seconds
>>>>>>> MEMLEAK:
>>>>>>> 5 MB/10sec
>>>>>>> 6 MB/10sec
>>>>>>> 4 MB/10sec
>>>>>>> 4 MB/10sec
>>>>>>>
>>>>>>>
>>>>>>> Other settings TSO/GRO off
>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>> for i in $ifc
>>>>>>> do
>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>> 512 tx-usecs 128
>>>>>>> ethtool -K $i gro off
>>>>>>> ethtool -K $i tso off
>>>>>>>
>>>>>>> done
>>>>>>>
>>>>>>> Same leak about 5MB per 10 seconds
>>>>>>> MEMLEAK:
>>>>>>> 5 MB/10sec
>>>>>>> 5 MB/10sec
>>>>>>> 5 MB/10sec
>>>>>>>
>>>>>>>
>>>>>>> Other settings rx-usecs change from 512 to 1024:
>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>> for i in $ifc
>>>>>>> do
>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>> 1024 tx-usecs 128
>>>>>>> ethtool -K $i gro off
>>>>>>> ethtool -K $i tso off
>>>>>>>
>>>>>>> done
>>>>>>>
>>>>>>> MEMLEAK:
>>>>>>> 4 MB/10sec
>>>>>>> 3 MB/10sec
>>>>>>> 4 MB/10sec
>>>>>>> 4 MB/10sec
>>>>>>>
>>>>>>>
>>>>>>> So memleak have something to do with rx-usecs (less interrupts
>>>>>>> but bigger latency for traffic)
>>>>>>>
>>>>>>>
>>>>>>> But also enabling TSO/GRO making leak about 1MB bigger for each
>>>>>>> 10 seconds
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> So far best config is:
>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>> enp3s0f2 enp3s0f3'
>>>>>> for i in $ifc
>>>>>> do
>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 64
>>>>>> tx-usecs 512
>>>>>> ethtool -K $i gro off
>>>>>> ethtool -K $i tso on
>>>>>>
>>>>>> done
>>>>>>
>>>>>> MEMLEAK - about 2MB/10secs
>>>>>> 2 MB/10sec
>>>>>> 2 MB/10sec
>>>>>> 2 MB/10sec
>>>>>>
>>>>>>
>>>>>> With - rx-usecs set to 256 (about 7-9MB/10secs memleak)
>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>> enp3s0f2 enp3s0f3'
>>>>>> for i in $ifc
>>>>>> do
>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>> 256 tx-usecs 512
>>>>>> ethtool -K $i gro off
>>>>>> ethtool -K $i tso on
>>>>>>
>>>>>> done
>>>>>>
>>>>>> MEMLEAK:
>>>>>> 7 MB/10sec
>>>>>> 7 MB/10sec
>>>>>> 8 MB/10sec
>>>>>> 9 MB/10sec
>>>>>>
>>>>>>
>>>>>
>>>>> And even less memleak with rx-usecs set to 32
>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>> enp3s0f2 enp3s0f3'
>>>>> for i in $ifc
>>>>> do
>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 32
>>>>> tx-usecs 512
>>>>> ethtool -K $i gro off
>>>>> ethtool -K $i tso on
>>>>>
>>>>> done
>>>>>
>>>>>
>>>>> MEMLEAK - about 0-2MB for each 10 seconds
>>>>> 0 MB/10sec
>>>>> 1 MB/10sec
>>>>> 0 MB/10sec
>>>>> 2 MB/10sec
>>>>> 1 MB/10sec
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> So best settings - to have as less leak as possible for now
>>>> (rx-usecs set to 16):
>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>> enp3s0f3'
>>>> for i in $ifc
>>>> do
>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 16
>>>> tx-usecs 768
>>>> ethtool -K $i gro on
>>>> ethtool -K $i tso on
>>>>
>>>> done
>>>>
>>>>
>>>> MEMLEAK: (0-1MB/10seconds)
>>>> 0 MB/10sec
>>>> 0 MB/10sec
>>>> 0 MB/10sec
>>>> 1 MB/10sec
>>>> 1 MB/10sec
>>>> -1 MB/10sec
>>>> 1 MB/10sec
>>>> 1 MB/10sec
>>>> 0 MB/10sec
>>>>
>>>> (there are some memory recycles - so this is good :) )
>>>>
>>>>
>>>>
>>>> Compared to(rx-usecs 512):
>>>>
>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>> enp3s0f3'
>>>> for i in $ifc
>>>> do
>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 512
>>>> tx-usecs 128
>>>> ethtool -K $i gro on
>>>> ethtool -K $i tso on
>>>>
>>>> done
>>>>
>>>> Server is leaking about 4-6MB per each 10 seconds
>>>> MEMLEAK:
>>>> 5 MB/10sec
>>>> 6 MB/10sec
>>>> 4 MB/10sec
>>>> 4 MB/10sec
>>>>
>>>>
>>>
>>> And graph where all changes for rx-usecs was done over some time:
>>> https://ibb.co/nrRfbR
>>>
>>>
>>>
>>>
>>>
>> Cant eliminate the problem with settings - memleak is bigger or less
>> visible with rx-usecs set to low values - but then have 100% cpu load
>> - cant have rx-usecs set to 16
>>
>> Cant find also other host with same cards or that are using i40e
>> driver for tests with bisecting
>> So will just replace to mellanox :)
>>
>>
> Also after fresh reboot with i40e
> startup settings:
> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
> enp3s0f3'
> for i in $ifc
> do
> ip link set up dev $i
> ethtool -A $i autoneg off rx off tx off
> ethtool -G $i rx 2048 tx 2048
> ip link set $i txqueuelen 1000
> #ethtool -C $i rx-usecs 256
> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17
> tx-usecs 125
> ethtool -L $i combined 6
> #ethtool -N $i rx-flow-hash udp4 sdfn
> #ethtool -K $i ntuple on
> #ethtool -K $i gro off
> #ethtool -K $i tso off
> done
>
>
> After issuing:
>
> ethtool -K enp2s0f0 gro on tso on
>
> dmesg shows
> [35764.338259] i40e 0000:02:00.0: PF reset failed, -15
>
>
> and no traffic on the card :)
>
>
Also checked now
bigger rx ring
ethtool -G $i rx 2048 tx 2048
Bigger memleag :)
^ permalink raw reply
* Re: Linux 4.12+ memory leak on router with i40e NICs
From: Paweł Staszewski @ 2017-10-18 22:58 UTC (permalink / raw)
To: Alexander Duyck
Cc: Pavlos Parissis, Anders K. Pedersen | Cohaesio,
netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org,
alexander.h.duyck@intel.com
In-Reply-To: <12670bc6-439c-7ef4-109a-fd20384b9ca2@itcare.pl>
W dniu 2017-10-19 o 00:50, Paweł Staszewski pisze:
>
>
> W dniu 2017-10-19 o 00:20, Paweł Staszewski pisze:
>>
>>
>> W dniu 2017-10-18 o 17:44, Paweł Staszewski pisze:
>>>
>>>
>>> W dniu 2017-10-17 o 16:08, Paweł Staszewski pisze:
>>>>
>>>>
>>>> W dniu 2017-10-17 o 13:52, Paweł Staszewski pisze:
>>>>>
>>>>>
>>>>> W dniu 2017-10-17 o 13:05, Paweł Staszewski pisze:
>>>>>>
>>>>>>
>>>>>> W dniu 2017-10-17 o 12:59, Paweł Staszewski pisze:
>>>>>>>
>>>>>>>
>>>>>>> W dniu 2017-10-17 o 12:51, Paweł Staszewski pisze:
>>>>>>>>
>>>>>>>>
>>>>>>>> W dniu 2017-10-17 o 12:20, Paweł Staszewski pisze:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> W dniu 2017-10-17 o 11:48, Paweł Staszewski pisze:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> W dniu 2017-10-17 o 02:44, Paweł Staszewski pisze:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> W dniu 2017-10-17 o 01:56, Alexander Duyck pisze:
>>>>>>>>>>>> On Mon, Oct 16, 2017 at 4:34 PM, Paweł Staszewski
>>>>>>>>>>>> <pstaszewski@itcare.pl> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> W dniu 2017-10-16 o 18:26, Paweł Staszewski pisze:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> W dniu 2017-10-16 o 13:20, Pavlos Parissis pisze:
>>>>>>>>>>>>>>> On 15/10/2017 02:58 πμ, Alexander Duyck wrote:
>>>>>>>>>>>>>>>> Hi Pawel,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To clarify is that Dave Miller's tree or Linus's that
>>>>>>>>>>>>>>>> you are talking
>>>>>>>>>>>>>>>> about? If it is Dave's tree how long ago was it you
>>>>>>>>>>>>>>>> pulled it since I
>>>>>>>>>>>>>>>> think the fix was just pushed by Jeff Kirsher a few
>>>>>>>>>>>>>>>> days ago.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The issue should be fixed in the following commit:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/drivers/net/ethernet/intel/i40e/i40e_txrx.c?id=2b9478ffc550f17c6cd8c69057234e91150f5972
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do you know when it is going to be available on net-next
>>>>>>>>>>>>>>> and linux-stable
>>>>>>>>>>>>>>> repos?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Pavlos
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I will make some tests today night with "net" git tree
>>>>>>>>>>>>>> where this patch is
>>>>>>>>>>>>>> included.
>>>>>>>>>>>>>> Starting from 0:00 CET
>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Upgraded and looks like problem is not solved with that patch
>>>>>>>>>>>>> Currently running system with
>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/
>>>>>>>>>>>>>
>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>
>>>>>>>>>>>>> Still about 0.5GB of memory is leaking somewhere
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also can confirm that the latest kernel where memory is
>>>>>>>>>>>>> not leaking (with
>>>>>>>>>>>>> use i40e driver intel 710 cards) is 4.11.12
>>>>>>>>>>>>> With kernel 4.11.12 - after hour no change in memory usage.
>>>>>>>>>>>>>
>>>>>>>>>>>>> also checked that with ixgbe instead of i40e with same
>>>>>>>>>>>>> net.git kernel there
>>>>>>>>>>>>> is no memleak - after hour same memory usage - so for 100%
>>>>>>>>>>>>> this is i40e
>>>>>>>>>>>>> driver problem.
>>>>>>>>>>>> So how long was the run to get the .5GB of memory leaking?
>>>>>>>>>>> 1 hour
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Also is there any chance of you being able to bisect to
>>>>>>>>>>>> determine
>>>>>>>>>>>> where the memory leak was introduced since as you pointed
>>>>>>>>>>>> out it
>>>>>>>>>>>> didn't exist in 4.11.12 so odds are it was introduced
>>>>>>>>>>>> somewhere
>>>>>>>>>>>> between 4.11 and the latest kernel release.
>>>>>>>>>>> Can be hard cause currently need to back to 4.11.12 - this
>>>>>>>>>>> is production host/router
>>>>>>>>>>> Will try to find some free/test router for tests/bicects
>>>>>>>>>>> with i40e driver (intel 710 cards)
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> - Alex
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Also forgoto to add errors for i40e when driver initialize:
>>>>>>>>>> [ 15.760569] i40e 0000:02:00.1: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>> [ 16.365587] i40e 0000:03:00.3: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>> [ 16.367686] i40e 0000:02:00.2: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>> [ 16.368816] i40e 0000:03:00.0: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>> [ 16.369877] i40e 0000:03:00.2: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>> [ 16.370941] i40e 0000:02:00.3: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>> [ 16.372005] i40e 0000:02:00.0: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>> [ 16.373029] i40e 0000:03:00.1: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>>
>>>>>>>>>> some params that are set for this nic's
>>>>>>>>>> ip link set up dev $i
>>>>>>>>>> ethtool -A $i autoneg off rx off tx off
>>>>>>>>>> ethtool -G $i rx 1024 tx 2048
>>>>>>>>>> ip link set $i txqueuelen 1000
>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>> rx-usecs 512 tx-usecs 128
>>>>>>>>>> ethtool -L $i combined 6
>>>>>>>>>> #ethtool -N $i rx-flow-hash udp4 sdfn
>>>>>>>>>> ethtool -K $i ntuple on
>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Also after TSO/GRO on there is memory usage change - and
>>>>>>>>> leaking faster
>>>>>>>>> Below image from memory usage before change with TSO/GRO OFF
>>>>>>>>> and after enabling TSO/GRO
>>>>>>>>>
>>>>>>>>> https://ibb.co/dTqBY6
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Pawel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> With settings like this:
>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>> for i in $ifc
>>>>>>>> do
>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>>> 512 tx-usecs 128
>>>>>>>> ethtool -K $i gro on
>>>>>>>> ethtool -K $i tso on
>>>>>>>>
>>>>>>>> done
>>>>>>>>
>>>>>>>> Server is leaking about 4-6MB per each 10 seconds
>>>>>>>> MEMLEAK:
>>>>>>>> 5 MB/10sec
>>>>>>>> 6 MB/10sec
>>>>>>>> 4 MB/10sec
>>>>>>>> 4 MB/10sec
>>>>>>>>
>>>>>>>>
>>>>>>>> Other settings TSO/GRO off
>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>> for i in $ifc
>>>>>>>> do
>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>>> 512 tx-usecs 128
>>>>>>>> ethtool -K $i gro off
>>>>>>>> ethtool -K $i tso off
>>>>>>>>
>>>>>>>> done
>>>>>>>>
>>>>>>>> Same leak about 5MB per 10 seconds
>>>>>>>> MEMLEAK:
>>>>>>>> 5 MB/10sec
>>>>>>>> 5 MB/10sec
>>>>>>>> 5 MB/10sec
>>>>>>>>
>>>>>>>>
>>>>>>>> Other settings rx-usecs change from 512 to 1024:
>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>> for i in $ifc
>>>>>>>> do
>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>>> 1024 tx-usecs 128
>>>>>>>> ethtool -K $i gro off
>>>>>>>> ethtool -K $i tso off
>>>>>>>>
>>>>>>>> done
>>>>>>>>
>>>>>>>> MEMLEAK:
>>>>>>>> 4 MB/10sec
>>>>>>>> 3 MB/10sec
>>>>>>>> 4 MB/10sec
>>>>>>>> 4 MB/10sec
>>>>>>>>
>>>>>>>>
>>>>>>>> So memleak have something to do with rx-usecs (less interrupts
>>>>>>>> but bigger latency for traffic)
>>>>>>>>
>>>>>>>>
>>>>>>>> But also enabling TSO/GRO making leak about 1MB bigger for each
>>>>>>>> 10 seconds
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> So far best config is:
>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>> for i in $ifc
>>>>>>> do
>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>> 64 tx-usecs 512
>>>>>>> ethtool -K $i gro off
>>>>>>> ethtool -K $i tso on
>>>>>>>
>>>>>>> done
>>>>>>>
>>>>>>> MEMLEAK - about 2MB/10secs
>>>>>>> 2 MB/10sec
>>>>>>> 2 MB/10sec
>>>>>>> 2 MB/10sec
>>>>>>>
>>>>>>>
>>>>>>> With - rx-usecs set to 256 (about 7-9MB/10secs memleak)
>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>> for i in $ifc
>>>>>>> do
>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>> 256 tx-usecs 512
>>>>>>> ethtool -K $i gro off
>>>>>>> ethtool -K $i tso on
>>>>>>>
>>>>>>> done
>>>>>>>
>>>>>>> MEMLEAK:
>>>>>>> 7 MB/10sec
>>>>>>> 7 MB/10sec
>>>>>>> 8 MB/10sec
>>>>>>> 9 MB/10sec
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> And even less memleak with rx-usecs set to 32
>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>> enp3s0f2 enp3s0f3'
>>>>>> for i in $ifc
>>>>>> do
>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 32
>>>>>> tx-usecs 512
>>>>>> ethtool -K $i gro off
>>>>>> ethtool -K $i tso on
>>>>>>
>>>>>> done
>>>>>>
>>>>>>
>>>>>> MEMLEAK - about 0-2MB for each 10 seconds
>>>>>> 0 MB/10sec
>>>>>> 1 MB/10sec
>>>>>> 0 MB/10sec
>>>>>> 2 MB/10sec
>>>>>> 1 MB/10sec
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> So best settings - to have as less leak as possible for now
>>>>> (rx-usecs set to 16):
>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>> enp3s0f2 enp3s0f3'
>>>>> for i in $ifc
>>>>> do
>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 16
>>>>> tx-usecs 768
>>>>> ethtool -K $i gro on
>>>>> ethtool -K $i tso on
>>>>>
>>>>> done
>>>>>
>>>>>
>>>>> MEMLEAK: (0-1MB/10seconds)
>>>>> 0 MB/10sec
>>>>> 0 MB/10sec
>>>>> 0 MB/10sec
>>>>> 1 MB/10sec
>>>>> 1 MB/10sec
>>>>> -1 MB/10sec
>>>>> 1 MB/10sec
>>>>> 1 MB/10sec
>>>>> 0 MB/10sec
>>>>>
>>>>> (there are some memory recycles - so this is good :) )
>>>>>
>>>>>
>>>>>
>>>>> Compared to(rx-usecs 512):
>>>>>
>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>> enp3s0f2 enp3s0f3'
>>>>> for i in $ifc
>>>>> do
>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 512
>>>>> tx-usecs 128
>>>>> ethtool -K $i gro on
>>>>> ethtool -K $i tso on
>>>>>
>>>>> done
>>>>>
>>>>> Server is leaking about 4-6MB per each 10 seconds
>>>>> MEMLEAK:
>>>>> 5 MB/10sec
>>>>> 6 MB/10sec
>>>>> 4 MB/10sec
>>>>> 4 MB/10sec
>>>>>
>>>>>
>>>>
>>>> And graph where all changes for rx-usecs was done over some time:
>>>> https://ibb.co/nrRfbR
>>>>
>>>>
>>>>
>>>>
>>>>
>>> Cant eliminate the problem with settings - memleak is bigger or less
>>> visible with rx-usecs set to low values - but then have 100% cpu
>>> load - cant have rx-usecs set to 16
>>>
>>> Cant find also other host with same cards or that are using i40e
>>> driver for tests with bisecting
>>> So will just replace to mellanox :)
>>>
>>>
>> Also after fresh reboot with i40e
>> startup settings:
>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>> enp3s0f3'
>> for i in $ifc
>> do
>> ip link set up dev $i
>> ethtool -A $i autoneg off rx off tx off
>> ethtool -G $i rx 2048 tx 2048
>> ip link set $i txqueuelen 1000
>> #ethtool -C $i rx-usecs 256
>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17
>> tx-usecs 125
>> ethtool -L $i combined 6
>> #ethtool -N $i rx-flow-hash udp4 sdfn
>> #ethtool -K $i ntuple on
>> #ethtool -K $i gro off
>> #ethtool -K $i tso off
>> done
>>
>>
>> After issuing:
>>
>> ethtool -K enp2s0f0 gro on tso on
>>
>> dmesg shows
>> [35764.338259] i40e 0000:02:00.0: PF reset failed, -15
>>
>>
>> and no traffic on the card :)
>>
>>
> Also checked now
> bigger rx ring
> ethtool -G $i rx 2048 tx 2048
>
>
> Bigger memleag :)
>
>
>
ok need to change cards now to ixgbe .... no reply no help for i40e so ....
maybee someone else with i40e will gather more data i have only this
host soo far - will try to install this cards to other hosts after
change but alll this movement will takes about 2 maybee 3 months -
nobody from my team want to but now cards that supports i40e cause of
this bug soo this is hard now to debug - i need to change also all cards
now >10G to mellanox that have no such bug ... sorry :)
^ permalink raw reply
* Re: [10/26] netfilter: Allow xt_owner in any user namespace
From: Andrei Vagin @ 2017-10-18 23:00 UTC (permalink / raw)
To: Pablo Neira Ayuso, Eric W. Biederman; +Cc: netfilter-devel, davem, netdev
In-Reply-To: <1467815048-2240-11-git-send-email-pablo@netfilter.org>
On Wed, Jul 06, 2016 at 04:23:52PM +0200, Pablo Neira Ayuso wrote:
> From: "Eric W. Biederman" <ebiederm@xmission.com>
>
> Making this work is a little tricky as it really isn't kosher to
> change the xt_owner_match_info in a check function.
>
> Without changing xt_owner_match_info we need to know the user
> namespace the uids and gids are specified in. In the common case
> net->user_ns == current_user_ns(). Verify net->user_ns ==
> current_user_ns() in owner_check so we can later assume it in
> owner_mt.
>
> In owner_check also verify that all of the uids and gids specified are
> in net->user_ns and that the expected min/max relationship exists
> between the uids and gids in xt_owner_match_info.
>
> In owner_mt get the network namespace from the outgoing socket, as this
> must be the same network namespace as the netfilter rules, and use that
> network namespace to find the user namespace the uids and gids in
> xt_match_owner_info are encoded in. Then convert from their encoded
> from into the kernel internal format for uids and gids and perform the
> owner match.
>
> Similar to ping_group_range, this code does not try to detect
> noncontiguous UID/GID ranges.
>
> Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
> Signed-off-by: Kevin Cernekee <cernekee@chromium.org>
> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
> ---
> net/netfilter/xt_owner.c | 41 +++++++++++++++++++++++++++++++++++------
> 1 file changed, 35 insertions(+), 6 deletions(-)
>
> diff --git a/net/netfilter/xt_owner.c b/net/netfilter/xt_owner.c
> index 1302b47..a20e731 100644
> --- a/net/netfilter/xt_owner.c
> +++ b/net/netfilter/xt_owner.c
> @@ -21,11 +21,39 @@
> static int owner_check(const struct xt_mtchk_param *par)
> {
> struct xt_owner_match_info *info = par->matchinfo;
> + struct net *net = par->net;
>
> - /* For now only allow adding matches from the initial user namespace */
> + /* Only allow the common case where the userns of the writer
> + * matches the userns of the network namespace.
> + */
> if ((info->match & (XT_OWNER_UID|XT_OWNER_GID)) &&
> - (current_user_ns() != &init_user_ns))
> + (current_user_ns() != net->user_ns))
> return -EINVAL;
Here is an issue. If we once added an xt_owner rule, we will not be able to
add/delete/change any other iptables rules from other user namespaces even from
init_user_ns.
We meet this issue in CRIU. We add a few rules to block all traffic in a
network namespace before dumping sockets:
[avagin@laptop ~]$ cat criu-netlock.nf
*filter
:CRIU - [0:0]
-I INPUT -j CRIU
-I OUTPUT -j CRIU
-A CRIU -j DROP
COMMIT
We execute iptables-restore in a target netns from init_user_ns, and it's
always worked for us before we meet xt_owner rules.
[avagin@laptop ~]$ unshare -Urn sh -c 'iptables -t filter -A OUTPUT -p tcp --dport 666 -m owner --uid-owner 0 -j DROP && echo $$; sleep 1000'
16375
^Z
[1]+ Stopped unshare -Urn sh -c 'iptables -t filter -A OUTPUT -p tcp --dport 666 -m owner --uid-owner 0 -j DROP && echo $$; sleep 1000'
[avagin@laptop ~]$ sudo nsenter -n -t 16375 -- /usr/sbin/iptables-restore -n < criu-netlock.nf
iptables-restore: line 6 failed
# now let's try to remove the xt_owner rule and check that everything works as expected
[avagin@laptop ~]$ sudo nsenter -n -t 16375 -- iptables -t filter -D OUTPUT -p tcp --dport 666 -m owner --uid-owner 0 -j DROP
[avagin@laptop ~]$ sudo nsenter -n -t 16375 -- /usr/sbin/iptables-restore -n < criu-netlock.nf
> +
> + /* Ensure the uids are valid */
> + if (info->match & XT_OWNER_UID) {
> + kuid_t uid_min = make_kuid(net->user_ns, info->uid_min);
> + kuid_t uid_max = make_kuid(net->user_ns, info->uid_max);
> +
> + if (!uid_valid(uid_min) || !uid_valid(uid_max) ||
> + (info->uid_max < info->uid_min) ||
> + uid_lt(uid_max, uid_min)) {
> + return -EINVAL;
> + }
> + }
> +
> + /* Ensure the gids are valid */
> + if (info->match & XT_OWNER_GID) {
> + kgid_t gid_min = make_kgid(net->user_ns, info->gid_min);
> + kgid_t gid_max = make_kgid(net->user_ns, info->gid_max);
> +
> + if (!gid_valid(gid_min) || !gid_valid(gid_max) ||
> + (info->gid_max < info->gid_min) ||
> + gid_lt(gid_max, gid_min)) {
> + return -EINVAL;
> + }
> + }
> +
> return 0;
> }
>
> @@ -35,6 +63,7 @@ owner_mt(const struct sk_buff *skb, struct xt_action_param *par)
> const struct xt_owner_match_info *info = par->matchinfo;
> const struct file *filp;
> struct sock *sk = skb_to_full_sk(skb);
> + struct net *net = par->net;
>
> if (sk == NULL || sk->sk_socket == NULL)
> return (info->match ^ info->invert) == 0;
> @@ -51,8 +80,8 @@ owner_mt(const struct sk_buff *skb, struct xt_action_param *par)
> (XT_OWNER_UID | XT_OWNER_GID)) == 0;
>
> if (info->match & XT_OWNER_UID) {
> - kuid_t uid_min = make_kuid(&init_user_ns, info->uid_min);
> - kuid_t uid_max = make_kuid(&init_user_ns, info->uid_max);
> + kuid_t uid_min = make_kuid(net->user_ns, info->uid_min);
> + kuid_t uid_max = make_kuid(net->user_ns, info->uid_max);
> if ((uid_gte(filp->f_cred->fsuid, uid_min) &&
> uid_lte(filp->f_cred->fsuid, uid_max)) ^
> !(info->invert & XT_OWNER_UID))
> @@ -60,8 +89,8 @@ owner_mt(const struct sk_buff *skb, struct xt_action_param *par)
> }
>
> if (info->match & XT_OWNER_GID) {
> - kgid_t gid_min = make_kgid(&init_user_ns, info->gid_min);
> - kgid_t gid_max = make_kgid(&init_user_ns, info->gid_max);
> + kgid_t gid_min = make_kgid(net->user_ns, info->gid_min);
> + kgid_t gid_max = make_kgid(net->user_ns, info->gid_max);
> if ((gid_gte(filp->f_cred->fsgid, gid_min) &&
> gid_lte(filp->f_cred->fsgid, gid_max)) ^
> !(info->invert & XT_OWNER_GID))
^ permalink raw reply
* [PATCH v2 net] packet: avoid panic in packet_getsockopt()
From: Eric Dumazet @ 2017-10-18 23:14 UTC (permalink / raw)
To: David Miller; +Cc: netdev, Willem de Bruijn, Eric Dumazet, John Sperbeck
In-Reply-To: <1508364504.31614.150.camel@edumazet-glaptop3.roam.corp.google.com>
From: Eric Dumazet <edumazet@google.com>
syzkaller got crashes in packet_getsockopt() processing
PACKET_ROLLOVER_STATS command while another thread was managing
to change po->rollover
Using RCU will fix this bug. We might later add proper RCU annotations
for sparse sake.
In v2: I replaced kfree(rollover) in fanout_add() to kfree_rcu()
variant, as spotted by John.
Fixes: a9b6391814d5 ("packet: rollover statistics")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Willem de Bruijn <willemb@google.com>
Cc: John Sperbeck <jsperbeck@google.com>
---
net/packet/af_packet.c | 24 ++++++++++++++++--------
1 file changed, 16 insertions(+), 8 deletions(-)
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index bec01a3daf5b02bd716dbff5c9efef8d6a7982be..2986941164b1952b3b6014ff81d2986b504c334a 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -1769,7 +1769,7 @@ static int fanout_add(struct sock *sk, u16 id, u16 type_flags)
out:
if (err && rollover) {
- kfree(rollover);
+ kfree_rcu(rollover, rcu);
po->rollover = NULL;
}
mutex_unlock(&fanout_mutex);
@@ -1796,8 +1796,10 @@ static struct packet_fanout *fanout_release(struct sock *sk)
else
f = NULL;
- if (po->rollover)
+ if (po->rollover) {
kfree_rcu(po->rollover, rcu);
+ po->rollover = NULL;
+ }
}
mutex_unlock(&fanout_mutex);
@@ -3851,6 +3853,7 @@ static int packet_getsockopt(struct socket *sock, int level, int optname,
void *data = &val;
union tpacket_stats_u st;
struct tpacket_rollover_stats rstats;
+ struct packet_rollover *rollover;
if (level != SOL_PACKET)
return -ENOPROTOOPT;
@@ -3929,13 +3932,18 @@ static int packet_getsockopt(struct socket *sock, int level, int optname,
0);
break;
case PACKET_ROLLOVER_STATS:
- if (!po->rollover)
+ rcu_read_lock();
+ rollover = rcu_dereference(po->rollover);
+ if (rollover) {
+ rstats.tp_all = atomic_long_read(&rollover->num);
+ rstats.tp_huge = atomic_long_read(&rollover->num_huge);
+ rstats.tp_failed = atomic_long_read(&rollover->num_failed);
+ data = &rstats;
+ lv = sizeof(rstats);
+ }
+ rcu_read_unlock();
+ if (!rollover)
return -EINVAL;
- rstats.tp_all = atomic_long_read(&po->rollover->num);
- rstats.tp_huge = atomic_long_read(&po->rollover->num_huge);
- rstats.tp_failed = atomic_long_read(&po->rollover->num_failed);
- data = &rstats;
- lv = sizeof(rstats);
break;
case PACKET_TX_HAS_OFF:
val = po->tp_tx_has_off;
^ permalink raw reply related
* Re: Linux 4.12+ memory leak on router with i40e NICs
From: Paweł Staszewski @ 2017-10-18 23:22 UTC (permalink / raw)
To: Alexander Duyck
Cc: Pavlos Parissis, Anders K. Pedersen | Cohaesio,
netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org,
alexander.h.duyck@intel.com
In-Reply-To: <bd59ccc3-2af4-6842-8d2f-169229a61470@itcare.pl>
W dniu 2017-10-19 o 00:58, Paweł Staszewski pisze:
>
>
> W dniu 2017-10-19 o 00:50, Paweł Staszewski pisze:
>>
>>
>> W dniu 2017-10-19 o 00:20, Paweł Staszewski pisze:
>>>
>>>
>>> W dniu 2017-10-18 o 17:44, Paweł Staszewski pisze:
>>>>
>>>>
>>>> W dniu 2017-10-17 o 16:08, Paweł Staszewski pisze:
>>>>>
>>>>>
>>>>> W dniu 2017-10-17 o 13:52, Paweł Staszewski pisze:
>>>>>>
>>>>>>
>>>>>> W dniu 2017-10-17 o 13:05, Paweł Staszewski pisze:
>>>>>>>
>>>>>>>
>>>>>>> W dniu 2017-10-17 o 12:59, Paweł Staszewski pisze:
>>>>>>>>
>>>>>>>>
>>>>>>>> W dniu 2017-10-17 o 12:51, Paweł Staszewski pisze:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> W dniu 2017-10-17 o 12:20, Paweł Staszewski pisze:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> W dniu 2017-10-17 o 11:48, Paweł Staszewski pisze:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> W dniu 2017-10-17 o 02:44, Paweł Staszewski pisze:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> W dniu 2017-10-17 o 01:56, Alexander Duyck pisze:
>>>>>>>>>>>>> On Mon, Oct 16, 2017 at 4:34 PM, Paweł Staszewski
>>>>>>>>>>>>> <pstaszewski@itcare.pl> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> W dniu 2017-10-16 o 18:26, Paweł Staszewski pisze:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> W dniu 2017-10-16 o 13:20, Pavlos Parissis pisze:
>>>>>>>>>>>>>>>> On 15/10/2017 02:58 πμ, Alexander Duyck wrote:
>>>>>>>>>>>>>>>>> Hi Pawel,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> To clarify is that Dave Miller's tree or Linus's that
>>>>>>>>>>>>>>>>> you are talking
>>>>>>>>>>>>>>>>> about? If it is Dave's tree how long ago was it you
>>>>>>>>>>>>>>>>> pulled it since I
>>>>>>>>>>>>>>>>> think the fix was just pushed by Jeff Kirsher a few
>>>>>>>>>>>>>>>>> days ago.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The issue should be fixed in the following commit:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/drivers/net/ethernet/intel/i40e/i40e_txrx.c?id=2b9478ffc550f17c6cd8c69057234e91150f5972
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Do you know when it is going to be available on
>>>>>>>>>>>>>>>> net-next and linux-stable
>>>>>>>>>>>>>>>> repos?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Pavlos
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I will make some tests today night with "net" git tree
>>>>>>>>>>>>>>> where this patch is
>>>>>>>>>>>>>>> included.
>>>>>>>>>>>>>>> Starting from 0:00 CET
>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Upgraded and looks like problem is not solved with that
>>>>>>>>>>>>>> patch
>>>>>>>>>>>>>> Currently running system with
>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Still about 0.5GB of memory is leaking somewhere
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also can confirm that the latest kernel where memory is
>>>>>>>>>>>>>> not leaking (with
>>>>>>>>>>>>>> use i40e driver intel 710 cards) is 4.11.12
>>>>>>>>>>>>>> With kernel 4.11.12 - after hour no change in memory usage.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> also checked that with ixgbe instead of i40e with same
>>>>>>>>>>>>>> net.git kernel there
>>>>>>>>>>>>>> is no memleak - after hour same memory usage - so for
>>>>>>>>>>>>>> 100% this is i40e
>>>>>>>>>>>>>> driver problem.
>>>>>>>>>>>>> So how long was the run to get the .5GB of memory leaking?
>>>>>>>>>>>> 1 hour
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also is there any chance of you being able to bisect to
>>>>>>>>>>>>> determine
>>>>>>>>>>>>> where the memory leak was introduced since as you pointed
>>>>>>>>>>>>> out it
>>>>>>>>>>>>> didn't exist in 4.11.12 so odds are it was introduced
>>>>>>>>>>>>> somewhere
>>>>>>>>>>>>> between 4.11 and the latest kernel release.
>>>>>>>>>>>> Can be hard cause currently need to back to 4.11.12 - this
>>>>>>>>>>>> is production host/router
>>>>>>>>>>>> Will try to find some free/test router for tests/bicects
>>>>>>>>>>>> with i40e driver (intel 710 cards)
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Alex
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Also forgoto to add errors for i40e when driver initialize:
>>>>>>>>>>> [ 15.760569] i40e 0000:02:00.1: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>>> [ 16.365587] i40e 0000:03:00.3: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>>> [ 16.367686] i40e 0000:02:00.2: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>>> [ 16.368816] i40e 0000:03:00.0: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>>> [ 16.369877] i40e 0000:03:00.2: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>>> [ 16.370941] i40e 0000:02:00.3: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>>> [ 16.372005] i40e 0000:02:00.0: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>>> [ 16.373029] i40e 0000:03:00.1: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>> adding RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>
>>>>>>>>>>> some params that are set for this nic's
>>>>>>>>>>> ip link set up dev $i
>>>>>>>>>>> ethtool -A $i autoneg off rx off tx off
>>>>>>>>>>> ethtool -G $i rx 1024 tx 2048
>>>>>>>>>>> ip link set $i txqueuelen 1000
>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>>> rx-usecs 512 tx-usecs 128
>>>>>>>>>>> ethtool -L $i combined 6
>>>>>>>>>>> #ethtool -N $i rx-flow-hash udp4 sdfn
>>>>>>>>>>> ethtool -K $i ntuple on
>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Also after TSO/GRO on there is memory usage change - and
>>>>>>>>>> leaking faster
>>>>>>>>>> Below image from memory usage before change with TSO/GRO OFF
>>>>>>>>>> and after enabling TSO/GRO
>>>>>>>>>>
>>>>>>>>>> https://ibb.co/dTqBY6
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Pawel
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> With settings like this:
>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>> for i in $ifc
>>>>>>>>> do
>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>>>> 512 tx-usecs 128
>>>>>>>>> ethtool -K $i gro on
>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>
>>>>>>>>> done
>>>>>>>>>
>>>>>>>>> Server is leaking about 4-6MB per each 10 seconds
>>>>>>>>> MEMLEAK:
>>>>>>>>> 5 MB/10sec
>>>>>>>>> 6 MB/10sec
>>>>>>>>> 4 MB/10sec
>>>>>>>>> 4 MB/10sec
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Other settings TSO/GRO off
>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>> for i in $ifc
>>>>>>>>> do
>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>>>> 512 tx-usecs 128
>>>>>>>>> ethtool -K $i gro off
>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>
>>>>>>>>> done
>>>>>>>>>
>>>>>>>>> Same leak about 5MB per 10 seconds
>>>>>>>>> MEMLEAK:
>>>>>>>>> 5 MB/10sec
>>>>>>>>> 5 MB/10sec
>>>>>>>>> 5 MB/10sec
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Other settings rx-usecs change from 512 to 1024:
>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>> for i in $ifc
>>>>>>>>> do
>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>>>> 1024 tx-usecs 128
>>>>>>>>> ethtool -K $i gro off
>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>
>>>>>>>>> done
>>>>>>>>>
>>>>>>>>> MEMLEAK:
>>>>>>>>> 4 MB/10sec
>>>>>>>>> 3 MB/10sec
>>>>>>>>> 4 MB/10sec
>>>>>>>>> 4 MB/10sec
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So memleak have something to do with rx-usecs (less interrupts
>>>>>>>>> but bigger latency for traffic)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> But also enabling TSO/GRO making leak about 1MB bigger for
>>>>>>>>> each 10 seconds
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> So far best config is:
>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>> for i in $ifc
>>>>>>>> do
>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>>> 64 tx-usecs 512
>>>>>>>> ethtool -K $i gro off
>>>>>>>> ethtool -K $i tso on
>>>>>>>>
>>>>>>>> done
>>>>>>>>
>>>>>>>> MEMLEAK - about 2MB/10secs
>>>>>>>> 2 MB/10sec
>>>>>>>> 2 MB/10sec
>>>>>>>> 2 MB/10sec
>>>>>>>>
>>>>>>>>
>>>>>>>> With - rx-usecs set to 256 (about 7-9MB/10secs memleak)
>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>> for i in $ifc
>>>>>>>> do
>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>>> 256 tx-usecs 512
>>>>>>>> ethtool -K $i gro off
>>>>>>>> ethtool -K $i tso on
>>>>>>>>
>>>>>>>> done
>>>>>>>>
>>>>>>>> MEMLEAK:
>>>>>>>> 7 MB/10sec
>>>>>>>> 7 MB/10sec
>>>>>>>> 8 MB/10sec
>>>>>>>> 9 MB/10sec
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> And even less memleak with rx-usecs set to 32
>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>> for i in $ifc
>>>>>>> do
>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>> 32 tx-usecs 512
>>>>>>> ethtool -K $i gro off
>>>>>>> ethtool -K $i tso on
>>>>>>>
>>>>>>> done
>>>>>>>
>>>>>>>
>>>>>>> MEMLEAK - about 0-2MB for each 10 seconds
>>>>>>> 0 MB/10sec
>>>>>>> 1 MB/10sec
>>>>>>> 0 MB/10sec
>>>>>>> 2 MB/10sec
>>>>>>> 1 MB/10sec
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> So best settings - to have as less leak as possible for now
>>>>>> (rx-usecs set to 16):
>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>> enp3s0f2 enp3s0f3'
>>>>>> for i in $ifc
>>>>>> do
>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 16
>>>>>> tx-usecs 768
>>>>>> ethtool -K $i gro on
>>>>>> ethtool -K $i tso on
>>>>>>
>>>>>> done
>>>>>>
>>>>>>
>>>>>> MEMLEAK: (0-1MB/10seconds)
>>>>>> 0 MB/10sec
>>>>>> 0 MB/10sec
>>>>>> 0 MB/10sec
>>>>>> 1 MB/10sec
>>>>>> 1 MB/10sec
>>>>>> -1 MB/10sec
>>>>>> 1 MB/10sec
>>>>>> 1 MB/10sec
>>>>>> 0 MB/10sec
>>>>>>
>>>>>> (there are some memory recycles - so this is good :) )
>>>>>>
>>>>>>
>>>>>>
>>>>>> Compared to(rx-usecs 512):
>>>>>>
>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>> enp3s0f2 enp3s0f3'
>>>>>> for i in $ifc
>>>>>> do
>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>> 512 tx-usecs 128
>>>>>> ethtool -K $i gro on
>>>>>> ethtool -K $i tso on
>>>>>>
>>>>>> done
>>>>>>
>>>>>> Server is leaking about 4-6MB per each 10 seconds
>>>>>> MEMLEAK:
>>>>>> 5 MB/10sec
>>>>>> 6 MB/10sec
>>>>>> 4 MB/10sec
>>>>>> 4 MB/10sec
>>>>>>
>>>>>>
>>>>>
>>>>> And graph where all changes for rx-usecs was done over some time:
>>>>> https://ibb.co/nrRfbR
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> Cant eliminate the problem with settings - memleak is bigger or
>>>> less visible with rx-usecs set to low values - but then have 100%
>>>> cpu load - cant have rx-usecs set to 16
>>>>
>>>> Cant find also other host with same cards or that are using i40e
>>>> driver for tests with bisecting
>>>> So will just replace to mellanox :)
>>>>
>>>>
>>> Also after fresh reboot with i40e
>>> startup settings:
>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>> enp3s0f3'
>>> for i in $ifc
>>> do
>>> ip link set up dev $i
>>> ethtool -A $i autoneg off rx off tx off
>>> ethtool -G $i rx 2048 tx 2048
>>> ip link set $i txqueuelen 1000
>>> #ethtool -C $i rx-usecs 256
>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17
>>> tx-usecs 125
>>> ethtool -L $i combined 6
>>> #ethtool -N $i rx-flow-hash udp4 sdfn
>>> #ethtool -K $i ntuple on
>>> #ethtool -K $i gro off
>>> #ethtool -K $i tso off
>>> done
>>>
>>>
>>> After issuing:
>>>
>>> ethtool -K enp2s0f0 gro on tso on
>>>
>>> dmesg shows
>>> [35764.338259] i40e 0000:02:00.0: PF reset failed, -15
>>>
>>>
>>> and no traffic on the card :)
>>>
>>>
>> Also checked now
>> bigger rx ring
>> ethtool -G $i rx 2048 tx 2048
>>
>>
>> Bigger memleag :)
>>
>>
>>
> ok need to change cards now to ixgbe .... no reply no help for i40e so
> ....
>
> maybee someone else with i40e will gather more data i have only this
> host soo far - will try to install this cards to other hosts after
> change but alll this movement will takes about 2 maybee 3 months -
> nobody from my team want to but now cards that supports i40e cause of
> this bug soo this is hard now to debug - i need to change also all
> cards now >10G to mellanox that have no such bug ... sorry :)
>
>
Last tests from my side:)
settings
ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
enp3s0f3'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 2048 tx 2048
ip link set $i txqueuelen 1000
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17
tx-usecs 125
ethtool -L $i combined 6
ethtool -K $i ntuple on
ethtool -K $i gro on
ethtool -K $i tso on
done
MEMLEAK 1-2MB/10secs
1 MB/10sec
2 MB/10sec
1 MB/10sec
2 MB/10sec
2 MB/10sec
2 MB/10sec
1 MB/10sec
2 MB/10sec
2 MB/10sec
2 MB/10sec
1 MB/10sec
2 MB/10sec
1 MB/10sec
1 MB/10sec
0 MB/10sec
2 MB/10sec
2 MB/10sec
0 MB/10sec
2 MB/10sec
5 MB/10sec
Change rx-usecs 16 tx usecs 16
ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
enp3s0f3'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 2048 tx 2048
ip link set $i txqueuelen 1000
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 16
tx-usecs 16
ethtool -L $i combined 6
ethtool -K $i ntuple on
ethtool -K $i gro on
ethtool -K $i tso on
done
MEMLEAK: 0-2MB/s with some recycles
0 MB/10sec
0 MB/10sec
0 MB/10sec
0 MB/10sec
0 MB/10sec
0 MB/10sec
1 MB/10sec
0 MB/10sec
2 MB/10sec
0 MB/10sec
2 MB/10sec
-1 MB/10sec
0 MB/10sec
2 MB/10sec
0 MB/10sec
2 MB/10sec
-1 MB/10sec
1 MB/10sec
^ permalink raw reply
* Re: Linux 4.12+ memory leak on router with i40e NICs
From: Alexander Duyck @ 2017-10-18 23:29 UTC (permalink / raw)
To: Paweł Staszewski, Pavlos Parissis, Alexander Duyck,
Anders K. Pedersen | Cohaesio, netdev@vger.kernel.org,
intel-wired-lan@lists.osuosl.org, alexander.h.duyck@intel.com
In-Reply-To: <20171017055155.GA19944@pc11.op.pod.cz>
On Mon, Oct 16, 2017 at 10:51 PM, Vitezslav Samel <vitezslav@samel.cz> wrote:
> On Tue, Oct 17, 2017 at 01:34:29AM +0200, Paweł Staszewski wrote:
>> W dniu 2017-10-16 o 18:26, Paweł Staszewski pisze:
>> > W dniu 2017-10-16 o 13:20, Pavlos Parissis pisze:
>> > > On 15/10/2017 02:58 πμ, Alexander Duyck wrote:
>> > > > Hi Pawel,
>> > > >
>> > > > To clarify is that Dave Miller's tree or Linus's that you are talking
>> > > > about? If it is Dave's tree how long ago was it you pulled it since I
>> > > > think the fix was just pushed by Jeff Kirsher a few days ago.
>> > > >
>> > > > The issue should be fixed in the following commit:
>> > > > https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/drivers/net/ethernet/intel/i40e/i40e_txrx.c?id=2b9478ffc550f17c6cd8c69057234e91150f5972
>> > >
>> > > Do you know when it is going to be available on net-next and
>> > > linux-stable repos?
>> > >
>> > > Cheers,
>> > > Pavlos
>> > >
>> > >
>> > I will make some tests today night with "net" git tree where this patch
>> > is included.
>> > Starting from 0:00 CET
>> > :)
>> >
>> >
>> Upgraded and looks like problem is not solved with that patch
>> Currently running system with
>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/
>> kernel
>>
>> Still about 0.5GB of memory is leaking somewhere
>>
>> Also can confirm that the latest kernel where memory is not leaking (with
>> use i40e driver intel 710 cards) is 4.11.12
>> With kernel 4.11.12 - after hour no change in memory usage.
>>
>> also checked that with ixgbe instead of i40e with same net.git kernel there
>> is no memleak - after hour same memory usage - so for 100% this is i40e
>> driver problem.
>
> I have (probably) the same problem here but with X520 cards: booting
> 4.12.x gives me oops after circa 20 minutes of our workload. Booting
> 4.9.y is OK. This machine is in production so any testing is very
> limited.
>
> Machine was stable for >2 months (on the desk before got to
> production) with 4.12.8 but with no traffic on X520 cards.
>
> Cheers,
>
> Vita
Sorry but it can't be the same issue since we are discussing a
different driver (i40e) running different hardware (X710 or XL170).
You might want to start a new thread for your issue, and/or if
possible file a bug on e1000.sf.net.
Thanks.
- Alex
^ permalink raw reply
* Re: Linux 4.12+ memory leak on router with i40e NICs
From: Alexander Duyck @ 2017-10-18 23:37 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Pavlos Parissis, Anders K. Pedersen | Cohaesio,
netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org,
alexander.h.duyck@intel.com
In-Reply-To: <888ec430-1d70-68e6-2ee8-bab53d21bb0f@itcare.pl>
On Wed, Oct 18, 2017 at 4:22 PM, Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>
> W dniu 2017-10-19 o 00:58, Paweł Staszewski pisze:
>
>>
>>
>> W dniu 2017-10-19 o 00:50, Paweł Staszewski pisze:
>>>
>>>
>>>
>>> W dniu 2017-10-19 o 00:20, Paweł Staszewski pisze:
>>>>
>>>>
>>>>
>>>> W dniu 2017-10-18 o 17:44, Paweł Staszewski pisze:
>>>>>
>>>>>
>>>>>
>>>>> W dniu 2017-10-17 o 16:08, Paweł Staszewski pisze:
>>>>>>
>>>>>>
>>>>>>
>>>>>> W dniu 2017-10-17 o 13:52, Paweł Staszewski pisze:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> W dniu 2017-10-17 o 13:05, Paweł Staszewski pisze:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> W dniu 2017-10-17 o 12:59, Paweł Staszewski pisze:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> W dniu 2017-10-17 o 12:51, Paweł Staszewski pisze:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> W dniu 2017-10-17 o 12:20, Paweł Staszewski pisze:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> W dniu 2017-10-17 o 11:48, Paweł Staszewski pisze:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> W dniu 2017-10-17 o 02:44, Paweł Staszewski pisze:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> W dniu 2017-10-17 o 01:56, Alexander Duyck pisze:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 16, 2017 at 4:34 PM, Paweł Staszewski
>>>>>>>>>>>>>> <pstaszewski@itcare.pl> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> W dniu 2017-10-16 o 18:26, Paweł Staszewski pisze:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> W dniu 2017-10-16 o 13:20, Pavlos Parissis pisze:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 15/10/2017 02:58 πμ, Alexander Duyck wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Pawel,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> To clarify is that Dave Miller's tree or Linus's that you
>>>>>>>>>>>>>>>>>> are talking
>>>>>>>>>>>>>>>>>> about? If it is Dave's tree how long ago was it you pulled
>>>>>>>>>>>>>>>>>> it since I
>>>>>>>>>>>>>>>>>> think the fix was just pushed by Jeff Kirsher a few days
>>>>>>>>>>>>>>>>>> ago.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The issue should be fixed in the following commit:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/drivers/net/ethernet/intel/i40e/i40e_txrx.c?id=2b9478ffc550f17c6cd8c69057234e91150f5972
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Do you know when it is going to be available on net-next
>>>>>>>>>>>>>>>>> and linux-stable
>>>>>>>>>>>>>>>>> repos?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Pavlos
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I will make some tests today night with "net" git tree where
>>>>>>>>>>>>>>>> this patch is
>>>>>>>>>>>>>>>> included.
>>>>>>>>>>>>>>>> Starting from 0:00 CET
>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Upgraded and looks like problem is not solved with that patch
>>>>>>>>>>>>>>> Currently running system with
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/
>>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Still about 0.5GB of memory is leaking somewhere
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also can confirm that the latest kernel where memory is not
>>>>>>>>>>>>>>> leaking (with
>>>>>>>>>>>>>>> use i40e driver intel 710 cards) is 4.11.12
>>>>>>>>>>>>>>> With kernel 4.11.12 - after hour no change in memory usage.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> also checked that with ixgbe instead of i40e with same
>>>>>>>>>>>>>>> net.git kernel there
>>>>>>>>>>>>>>> is no memleak - after hour same memory usage - so for 100%
>>>>>>>>>>>>>>> this is i40e
>>>>>>>>>>>>>>> driver problem.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So how long was the run to get the .5GB of memory leaking?
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1 hour
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also is there any chance of you being able to bisect to
>>>>>>>>>>>>>> determine
>>>>>>>>>>>>>> where the memory leak was introduced since as you pointed out
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> didn't exist in 4.11.12 so odds are it was introduced
>>>>>>>>>>>>>> somewhere
>>>>>>>>>>>>>> between 4.11 and the latest kernel release.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can be hard cause currently need to back to 4.11.12 - this is
>>>>>>>>>>>>> production host/router
>>>>>>>>>>>>> Will try to find some free/test router for tests/bicects with
>>>>>>>>>>>>> i40e driver (intel 710 cards)
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Alex
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Also forgoto to add errors for i40e when driver initialize:
>>>>>>>>>>>> [ 15.760569] i40e 0000:02:00.1: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>> [ 16.365587] i40e 0000:03:00.3: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>> [ 16.367686] i40e 0000:02:00.2: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>> [ 16.368816] i40e 0000:03:00.0: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>> [ 16.369877] i40e 0000:03:00.2: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>> [ 16.370941] i40e 0000:02:00.3: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>> [ 16.372005] i40e 0000:02:00.0: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>> [ 16.373029] i40e 0000:03:00.1: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>
>>>>>>>>>>>> some params that are set for this nic's
>>>>>>>>>>>> ip link set up dev $i
>>>>>>>>>>>> ethtool -A $i autoneg off rx off tx off
>>>>>>>>>>>> ethtool -G $i rx 1024 tx 2048
>>>>>>>>>>>> ip link set $i txqueuelen 1000
>>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>>>>>>> 512 tx-usecs 128
>>>>>>>>>>>> ethtool -L $i combined 6
>>>>>>>>>>>> #ethtool -N $i rx-flow-hash udp4 sdfn
>>>>>>>>>>>> ethtool -K $i ntuple on
>>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Also after TSO/GRO on there is memory usage change - and leaking
>>>>>>>>>>> faster
>>>>>>>>>>> Below image from memory usage before change with TSO/GRO OFF and
>>>>>>>>>>> after enabling TSO/GRO
>>>>>>>>>>>
>>>>>>>>>>> https://ibb.co/dTqBY6
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Pawel
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> With settings like this:
>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>>> for i in $ifc
>>>>>>>>>> do
>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 512
>>>>>>>>>> tx-usecs 128
>>>>>>>>>> ethtool -K $i gro on
>>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>>
>>>>>>>>>> done
>>>>>>>>>>
>>>>>>>>>> Server is leaking about 4-6MB per each 10 seconds
>>>>>>>>>> MEMLEAK:
>>>>>>>>>> 5 MB/10sec
>>>>>>>>>> 6 MB/10sec
>>>>>>>>>> 4 MB/10sec
>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Other settings TSO/GRO off
>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>>> for i in $ifc
>>>>>>>>>> do
>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 512
>>>>>>>>>> tx-usecs 128
>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>>
>>>>>>>>>> done
>>>>>>>>>>
>>>>>>>>>> Same leak about 5MB per 10 seconds
>>>>>>>>>> MEMLEAK:
>>>>>>>>>> 5 MB/10sec
>>>>>>>>>> 5 MB/10sec
>>>>>>>>>> 5 MB/10sec
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Other settings rx-usecs change from 512 to 1024:
>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>>> for i in $ifc
>>>>>>>>>> do
>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>>>>> 1024 tx-usecs 128
>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>>
>>>>>>>>>> done
>>>>>>>>>>
>>>>>>>>>> MEMLEAK:
>>>>>>>>>> 4 MB/10sec
>>>>>>>>>> 3 MB/10sec
>>>>>>>>>> 4 MB/10sec
>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> So memleak have something to do with rx-usecs (less interrupts but
>>>>>>>>>> bigger latency for traffic)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> But also enabling TSO/GRO making leak about 1MB bigger for each 10
>>>>>>>>>> seconds
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> So far best config is:
>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>>>>>>> enp3s0f3'
>>>>>>>>> for i in $ifc
>>>>>>>>> do
>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 64
>>>>>>>>> tx-usecs 512
>>>>>>>>> ethtool -K $i gro off
>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>
>>>>>>>>> done
>>>>>>>>>
>>>>>>>>> MEMLEAK - about 2MB/10secs
>>>>>>>>> 2 MB/10sec
>>>>>>>>> 2 MB/10sec
>>>>>>>>> 2 MB/10sec
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> With - rx-usecs set to 256 (about 7-9MB/10secs memleak)
>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>>>>>>> enp3s0f3'
>>>>>>>>> for i in $ifc
>>>>>>>>> do
>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 256
>>>>>>>>> tx-usecs 512
>>>>>>>>> ethtool -K $i gro off
>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>
>>>>>>>>> done
>>>>>>>>>
>>>>>>>>> MEMLEAK:
>>>>>>>>> 7 MB/10sec
>>>>>>>>> 7 MB/10sec
>>>>>>>>> 8 MB/10sec
>>>>>>>>> 9 MB/10sec
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> And even less memleak with rx-usecs set to 32
>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>>>>>> enp3s0f3'
>>>>>>>> for i in $ifc
>>>>>>>> do
>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 32
>>>>>>>> tx-usecs 512
>>>>>>>> ethtool -K $i gro off
>>>>>>>> ethtool -K $i tso on
>>>>>>>>
>>>>>>>> done
>>>>>>>>
>>>>>>>>
>>>>>>>> MEMLEAK - about 0-2MB for each 10 seconds
>>>>>>>> 0 MB/10sec
>>>>>>>> 1 MB/10sec
>>>>>>>> 0 MB/10sec
>>>>>>>> 2 MB/10sec
>>>>>>>> 1 MB/10sec
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So best settings - to have as less leak as possible for now (rx-usecs
>>>>>>> set to 16):
>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>>>>> enp3s0f3'
>>>>>>> for i in $ifc
>>>>>>> do
>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 16
>>>>>>> tx-usecs 768
>>>>>>> ethtool -K $i gro on
>>>>>>> ethtool -K $i tso on
>>>>>>>
>>>>>>> done
>>>>>>>
>>>>>>>
>>>>>>> MEMLEAK: (0-1MB/10seconds)
>>>>>>> 0 MB/10sec
>>>>>>> 0 MB/10sec
>>>>>>> 0 MB/10sec
>>>>>>> 1 MB/10sec
>>>>>>> 1 MB/10sec
>>>>>>> -1 MB/10sec
>>>>>>> 1 MB/10sec
>>>>>>> 1 MB/10sec
>>>>>>> 0 MB/10sec
>>>>>>>
>>>>>>> (there are some memory recycles - so this is good :) )
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Compared to(rx-usecs 512):
>>>>>>>
>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>>>>> enp3s0f3'
>>>>>>> for i in $ifc
>>>>>>> do
>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 512
>>>>>>> tx-usecs 128
>>>>>>> ethtool -K $i gro on
>>>>>>> ethtool -K $i tso on
>>>>>>>
>>>>>>> done
>>>>>>>
>>>>>>> Server is leaking about 4-6MB per each 10 seconds
>>>>>>> MEMLEAK:
>>>>>>> 5 MB/10sec
>>>>>>> 6 MB/10sec
>>>>>>> 4 MB/10sec
>>>>>>> 4 MB/10sec
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> And graph where all changes for rx-usecs was done over some time:
>>>>>> https://ibb.co/nrRfbR
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> Cant eliminate the problem with settings - memleak is bigger or less
>>>>> visible with rx-usecs set to low values - but then have 100% cpu load - cant
>>>>> have rx-usecs set to 16
>>>>>
>>>>> Cant find also other host with same cards or that are using i40e driver
>>>>> for tests with bisecting
>>>>> So will just replace to mellanox :)
>>>>>
>>>>>
>>>> Also after fresh reboot with i40e
>>>> startup settings:
>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>> enp3s0f3'
>>>> for i in $ifc
>>>> do
>>>> ip link set up dev $i
>>>> ethtool -A $i autoneg off rx off tx off
>>>> ethtool -G $i rx 2048 tx 2048
>>>> ip link set $i txqueuelen 1000
>>>> #ethtool -C $i rx-usecs 256
>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17
>>>> tx-usecs 125
>>>> ethtool -L $i combined 6
>>>> #ethtool -N $i rx-flow-hash udp4 sdfn
>>>> #ethtool -K $i ntuple on
>>>> #ethtool -K $i gro off
>>>> #ethtool -K $i tso off
>>>> done
>>>>
>>>>
>>>> After issuing:
>>>>
>>>> ethtool -K enp2s0f0 gro on tso on
>>>>
>>>> dmesg shows
>>>> [35764.338259] i40e 0000:02:00.0: PF reset failed, -15
>>>>
>>>>
>>>> and no traffic on the card :)
>>>>
>>>>
>>> Also checked now
>>> bigger rx ring
>>> ethtool -G $i rx 2048 tx 2048
>>>
>>>
>>> Bigger memleag :)
>>>
>>>
>>>
>> ok need to change cards now to ixgbe .... no reply no help for i40e so
>> ....
>>
>> maybee someone else with i40e will gather more data i have only this host
>> soo far - will try to install this cards to other hosts after change but
>> alll this movement will takes about 2 maybee 3 months - nobody from my team
>> want to but now cards that supports i40e cause of this bug soo this is hard
>> now to debug - i need to change also all cards now >10G to mellanox that
>> have no such bug ... sorry :)
>>
>>
> Last tests from my side:)
> settings
> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
> enp3s0f3'
> for i in $ifc
> do
> ip link set up dev $i
> ethtool -A $i autoneg off rx off tx off
> ethtool -G $i rx 2048 tx 2048
> ip link set $i txqueuelen 1000
> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17 tx-usecs
> 125
> ethtool -L $i combined 6
> ethtool -K $i ntuple on
> ethtool -K $i gro on
> ethtool -K $i tso on
> done
>
> MEMLEAK 1-2MB/10secs
> 1 MB/10sec
> 2 MB/10sec
> 1 MB/10sec
> 2 MB/10sec
> 2 MB/10sec
> 2 MB/10sec
> 1 MB/10sec
> 2 MB/10sec
> 2 MB/10sec
> 2 MB/10sec
> 1 MB/10sec
> 2 MB/10sec
> 1 MB/10sec
> 1 MB/10sec
> 0 MB/10sec
> 2 MB/10sec
> 2 MB/10sec
> 0 MB/10sec
> 2 MB/10sec
> 5 MB/10sec
>
> Change rx-usecs 16 tx usecs 16
> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
> enp3s0f3'
> for i in $ifc
> do
> ip link set up dev $i
> ethtool -A $i autoneg off rx off tx off
> ethtool -G $i rx 2048 tx 2048
> ip link set $i txqueuelen 1000
> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 16 tx-usecs
> 16
> ethtool -L $i combined 6
> ethtool -K $i ntuple on
> ethtool -K $i gro on
> ethtool -K $i tso on
> done
>
> MEMLEAK: 0-2MB/s with some recycles
> 0 MB/10sec
> 0 MB/10sec
> 0 MB/10sec
> 0 MB/10sec
> 0 MB/10sec
> 0 MB/10sec
> 1 MB/10sec
> 0 MB/10sec
> 2 MB/10sec
> 0 MB/10sec
> 2 MB/10sec
> -1 MB/10sec
> 0 MB/10sec
> 2 MB/10sec
> 0 MB/10sec
> 2 MB/10sec
> -1 MB/10sec
> 1 MB/10sec
This data doesn't tell me much of anything and isn't what I asked for.
I don't see how the interrupt throttling rate would be associated with
your memory leak other than possibly rate limiting it by rate limiting
the traffic itself. Is there something that gave you the impression
that interrupt rate was somehow involved?
When we last talked I had asked if you could do a git bisect to find
the memory leak and you said you would look into it. The most useful
way to solve this would be to do a git bisect between your current
kernel and the 4.11 kernel to find the point at which this started. If
we can do that then fixing this becomes much simpler as we just have
to fix the patch that introduced the issue.
Also, I don't know it is you are using to determine that there is a
memory leak. What tool is it you are using to do the tracking? Is
there any specific form of traffic that is causing the leak? If you
can't perform the bisection, any information you could provide that
would allow me to do it would also be useful.
Thanks.
- Alex
^ permalink raw reply
* Re: [PATCH net-next] tcp: socket option to set TCP fast open key
From: Christoph Paasch @ 2017-10-18 23:39 UTC (permalink / raw)
To: Yuchung Cheng; +Cc: davem, netdev, edumazet, ncardwell, maheshb
In-Reply-To: <20171018182251.5486-1-ycheng@google.com>
On 18/10/17 - 11:22:51, Yuchung Cheng wrote:
> New socket option TCP_FASTOPEN_KEY to allow different keys per
> listener. The listener by default uses the global key until the
> socket option is set. The key is a 16 bytes long binary data. This
> option has no effect on regular non-listener TCP sockets.
>
> Signed-off-by: Yuchung Cheng <ycheng@google.com>
> Reviewed-by: Eric Dumazet <edumazet@google.com>
> ---
> include/net/request_sock.h | 2 ++
> include/net/tcp.h | 5 +++--
> include/uapi/linux/tcp.h | 1 +
> net/ipv4/sysctl_net_ipv4.c | 3 ++-
> net/ipv4/tcp.c | 33 +++++++++++++++++++++++++++
> net/ipv4/tcp_fastopen.c | 56 +++++++++++++++++++++++++++++++++-------------
> net/ipv4/tcp_ipv4.c | 1 +
> 7 files changed, 82 insertions(+), 19 deletions(-)
Reviewed-by: Christoph Paasch <cpaasch@apple.com>
Christoph
^ permalink raw reply
* Re: Linux 4.12+ memory leak on router with i40e NICs
From: Paweł Staszewski @ 2017-10-18 23:40 UTC (permalink / raw)
To: Alexander Duyck, Pavlos Parissis, Anders K. Pedersen | Cohaesio,
netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org,
alexander.h.duyck@intel.com
In-Reply-To: <CAKgT0Udb5DVu6CB5U7rGNERZwMCKZUTii=qxn6hFzc=5zEZc1w@mail.gmail.com>
W dniu 2017-10-19 o 01:29, Alexander Duyck pisze:
> On Mon, Oct 16, 2017 at 10:51 PM, Vitezslav Samel <vitezslav@samel.cz> wrote:
>> On Tue, Oct 17, 2017 at 01:34:29AM +0200, Paweł Staszewski wrote:
>>> W dniu 2017-10-16 o 18:26, Paweł Staszewski pisze:
>>>> W dniu 2017-10-16 o 13:20, Pavlos Parissis pisze:
>>>>> On 15/10/2017 02:58 πμ, Alexander Duyck wrote:
>>>>>> Hi Pawel,
>>>>>>
>>>>>> To clarify is that Dave Miller's tree or Linus's that you are talking
>>>>>> about? If it is Dave's tree how long ago was it you pulled it since I
>>>>>> think the fix was just pushed by Jeff Kirsher a few days ago.
>>>>>>
>>>>>> The issue should be fixed in the following commit:
>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/drivers/net/ethernet/intel/i40e/i40e_txrx.c?id=2b9478ffc550f17c6cd8c69057234e91150f5972
>>>>> Do you know when it is going to be available on net-next and
>>>>> linux-stable repos?
>>>>>
>>>>> Cheers,
>>>>> Pavlos
>>>>>
>>>>>
>>>> I will make some tests today night with "net" git tree where this patch
>>>> is included.
>>>> Starting from 0:00 CET
>>>> :)
>>>>
>>>>
>>> Upgraded and looks like problem is not solved with that patch
>>> Currently running system with
>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/
>>> kernel
>>>
>>> Still about 0.5GB of memory is leaking somewhere
>>>
>>> Also can confirm that the latest kernel where memory is not leaking (with
>>> use i40e driver intel 710 cards) is 4.11.12
>>> With kernel 4.11.12 - after hour no change in memory usage.
>>>
>>> also checked that with ixgbe instead of i40e with same net.git kernel there
>>> is no memleak - after hour same memory usage - so for 100% this is i40e
>>> driver problem.
>> I have (probably) the same problem here but with X520 cards: booting
>> 4.12.x gives me oops after circa 20 minutes of our workload. Booting
>> 4.9.y is OK. This machine is in production so any testing is very
>> limited.
>>
>> Machine was stable for >2 months (on the desk before got to
>> production) with 4.12.8 but with no traffic on X520 cards.
>>
>> Cheers,
>>
>> Vita
> Sorry but it can't be the same issue since we are discussing a
> different driver (i40e) running different hardware (X710 or XL170).
> You might want to start a new thread for your issue, and/or if
> possible file a bug on e1000.sf.net.
>
> Thanks.
>
> - Alex
>
sorry but bugs reported on e1000.sf.net are delayed - some after about 6
or more months - when i reported first bug there iv got reply after a
year about no activity :):) haha - and reported there bug is still
actrive :)
better for me is now to change nics (for sure cheaper from the
perspective of clients :) ) to mellanox or just to replace and use ixgbe
- that have no this bug (mellanox and ixgbe have no such bug - have many
servers with them with same conf - and only one with i40e where is same
conf and memleak)
If nobody from Intel wants to reproduce this - qool - this is not my
problem but intels :) - there is now many good nics to use - like
mellanox or just stick with many 10G based on ixgbe that is really good
driver - but really ? intel guys have no XL710 cards ? i dont want to
buy another buggy cards to do only kernel bisects .... sorry ....
To do good bisects with this bug You need to spend maybee 200/300
bisects - and to confirm each - You need maybee 30minutes so count how
much time You need - more that 100 cards in price from mellanox maybee :)
so imagine what i will do :)
Thanks
Paweł
^ permalink raw reply
* Re: RFC(v2): Audit Kernel Container IDs
From: Aleksa Sarai @ 2017-10-18 23:46 UTC (permalink / raw)
To: Paul Moore, James Bottomley
Cc: mszeredi-H+wXaHxf7aLQT0dZR+AlfA, trondmy-7I+n7zu2hftEKMMhf/gKZA,
Simo Sorce, jlayton-H+wXaHxf7aLQT0dZR+AlfA, Carlos O'Donell,
API, Linux Containers, Linux Kernel, Eric Paris, David Howells,
Casey Schaufler, linux-audit-H+wXaHxf7aLQT0dZR+AlfA, Viro,
Andy Lutomirski, Development, Linux FS Devel,
cgroups-u79uwXL29TY76Z2rM5mHXA, Steve Grubb, Eric W. Biederman
In-Reply-To: <CAHC9VhRV9m6-APj3ofMQc22rL-WUoDzB8-urUxryszjCHHHLTg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
>> The security implications are that anything that can change the label
>> could also hide itself and its doings from the audit system and thus
>> would be used as a means to evade detection. I actually think this
>> means the label should be write once (once you've set it, you can't
>> change it) ...
>
> Richard and I have talked about a write once approach, but the
> thinking was that you may want to allow a nested container
> orchestrator (Why? I don't know, but people always want to do the
> craziest things.) and a write-once policy makes that impossible. If
> we punt on the nested orchestrator, I believe we can seriously think
> about a write-once policy to simplify things.
Nested containers are a very widely used use-case (see LXC system
containers, inside of which people run other container runtimes). So I
would definitely consider it something that "needs to be supported in
some way". While the LXC guys might be a *tad* crazy, the use-case isn't. :P
>> ... and orchestration systems should begin as unlabelled
>> processes allowing them to do arbitrary forks.
>
> My current thinking is that the default state is to start unlabeled (I
> just vomited a little into my SELinux hat); in other words
> init/systemd/PID-1 in the host system starts with an "unset" audit
> container ID. This not only helps define the host system (anything
> that has an unset audit container ID) but provides a blank slate for
> the orchestrator(s).
>
>> For nested containers, I actually think the label should be
>> hierarchical, so you can add a label for the new nested container but
>> it still also contains its parents label as well.
>
> I haven't made up my mind on this completely just yet, but I'm
> currently of the mindset that supporting multiple audit container IDs
> on a given process is not a good idea.
As long as creating a new "container" (that is, changing a process's
"audit container ID") is an audit event then I think that having a
hierarchy be explicit is not necessary (userspace audit can figure out
the hierarchy quite easily -- but also there are cases where thinking of
it as being hierarchical isn't necessarily correct).
--
Aleksa Sarai
Senior Software Engineer (Containers)
SUSE Linux GmbH
https://www.cyphar.com/
^ permalink raw reply
* Re: Linux 4.12+ memory leak on router with i40e NICs
From: Paweł Staszewski @ 2017-10-18 23:51 UTC (permalink / raw)
To: Alexander Duyck
Cc: Pavlos Parissis, Anders K. Pedersen | Cohaesio,
netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org,
alexander.h.duyck@intel.com
In-Reply-To: <CAKgT0Ud=GRrsNtoM0gOgzsh1uPJ7nuT9S-xbigmbUMi+iz-ufw@mail.gmail.com>
W dniu 2017-10-19 o 01:37, Alexander Duyck pisze:
> On Wed, Oct 18, 2017 at 4:22 PM, Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>>
>> W dniu 2017-10-19 o 00:58, Paweł Staszewski pisze:
>>
>>>
>>> W dniu 2017-10-19 o 00:50, Paweł Staszewski pisze:
>>>>
>>>>
>>>> W dniu 2017-10-19 o 00:20, Paweł Staszewski pisze:
>>>>>
>>>>>
>>>>> W dniu 2017-10-18 o 17:44, Paweł Staszewski pisze:
>>>>>>
>>>>>>
>>>>>> W dniu 2017-10-17 o 16:08, Paweł Staszewski pisze:
>>>>>>>
>>>>>>>
>>>>>>> W dniu 2017-10-17 o 13:52, Paweł Staszewski pisze:
>>>>>>>>
>>>>>>>>
>>>>>>>> W dniu 2017-10-17 o 13:05, Paweł Staszewski pisze:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> W dniu 2017-10-17 o 12:59, Paweł Staszewski pisze:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> W dniu 2017-10-17 o 12:51, Paweł Staszewski pisze:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> W dniu 2017-10-17 o 12:20, Paweł Staszewski pisze:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> W dniu 2017-10-17 o 11:48, Paweł Staszewski pisze:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> W dniu 2017-10-17 o 02:44, Paweł Staszewski pisze:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> W dniu 2017-10-17 o 01:56, Alexander Duyck pisze:
>>>>>>>>>>>>>>> On Mon, Oct 16, 2017 at 4:34 PM, Paweł Staszewski
>>>>>>>>>>>>>>> <pstaszewski@itcare.pl> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> W dniu 2017-10-16 o 18:26, Paweł Staszewski pisze:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> W dniu 2017-10-16 o 13:20, Pavlos Parissis pisze:
>>>>>>>>>>>>>>>>>> On 15/10/2017 02:58 πμ, Alexander Duyck wrote:
>>>>>>>>>>>>>>>>>>> Hi Pawel,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> To clarify is that Dave Miller's tree or Linus's that you
>>>>>>>>>>>>>>>>>>> are talking
>>>>>>>>>>>>>>>>>>> about? If it is Dave's tree how long ago was it you pulled
>>>>>>>>>>>>>>>>>>> it since I
>>>>>>>>>>>>>>>>>>> think the fix was just pushed by Jeff Kirsher a few days
>>>>>>>>>>>>>>>>>>> ago.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The issue should be fixed in the following commit:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/drivers/net/ethernet/intel/i40e/i40e_txrx.c?id=2b9478ffc550f17c6cd8c69057234e91150f5972
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Do you know when it is going to be available on net-next
>>>>>>>>>>>>>>>>>> and linux-stable
>>>>>>>>>>>>>>>>>> repos?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>> Pavlos
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I will make some tests today night with "net" git tree where
>>>>>>>>>>>>>>>>> this patch is
>>>>>>>>>>>>>>>>> included.
>>>>>>>>>>>>>>>>> Starting from 0:00 CET
>>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Upgraded and looks like problem is not solved with that patch
>>>>>>>>>>>>>>>> Currently running system with
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/
>>>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Still about 0.5GB of memory is leaking somewhere
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also can confirm that the latest kernel where memory is not
>>>>>>>>>>>>>>>> leaking (with
>>>>>>>>>>>>>>>> use i40e driver intel 710 cards) is 4.11.12
>>>>>>>>>>>>>>>> With kernel 4.11.12 - after hour no change in memory usage.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> also checked that with ixgbe instead of i40e with same
>>>>>>>>>>>>>>>> net.git kernel there
>>>>>>>>>>>>>>>> is no memleak - after hour same memory usage - so for 100%
>>>>>>>>>>>>>>>> this is i40e
>>>>>>>>>>>>>>>> driver problem.
>>>>>>>>>>>>>>> So how long was the run to get the .5GB of memory leaking?
>>>>>>>>>>>>>> 1 hour
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also is there any chance of you being able to bisect to
>>>>>>>>>>>>>>> determine
>>>>>>>>>>>>>>> where the memory leak was introduced since as you pointed out
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> didn't exist in 4.11.12 so odds are it was introduced
>>>>>>>>>>>>>>> somewhere
>>>>>>>>>>>>>>> between 4.11 and the latest kernel release.
>>>>>>>>>>>>>> Can be hard cause currently need to back to 4.11.12 - this is
>>>>>>>>>>>>>> production host/router
>>>>>>>>>>>>>> Will try to find some free/test router for tests/bicects with
>>>>>>>>>>>>>> i40e driver (intel 710 cards)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Alex
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Also forgoto to add errors for i40e when driver initialize:
>>>>>>>>>>>>> [ 15.760569] i40e 0000:02:00.1: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>> [ 16.365587] i40e 0000:03:00.3: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>> [ 16.367686] i40e 0000:02:00.2: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>> [ 16.368816] i40e 0000:03:00.0: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>> [ 16.369877] i40e 0000:03:00.2: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>> [ 16.370941] i40e 0000:02:00.3: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>> [ 16.372005] i40e 0000:02:00.0: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>> [ 16.373029] i40e 0000:03:00.1: Error I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>
>>>>>>>>>>>>> some params that are set for this nic's
>>>>>>>>>>>>> ip link set up dev $i
>>>>>>>>>>>>> ethtool -A $i autoneg off rx off tx off
>>>>>>>>>>>>> ethtool -G $i rx 1024 tx 2048
>>>>>>>>>>>>> ip link set $i txqueuelen 1000
>>>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>>>>>>>> 512 tx-usecs 128
>>>>>>>>>>>>> ethtool -L $i combined 6
>>>>>>>>>>>>> #ethtool -N $i rx-flow-hash udp4 sdfn
>>>>>>>>>>>>> ethtool -K $i ntuple on
>>>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Also after TSO/GRO on there is memory usage change - and leaking
>>>>>>>>>>>> faster
>>>>>>>>>>>> Below image from memory usage before change with TSO/GRO OFF and
>>>>>>>>>>>> after enabling TSO/GRO
>>>>>>>>>>>>
>>>>>>>>>>>> https://ibb.co/dTqBY6
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Pawel
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> With settings like this:
>>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>>>> for i in $ifc
>>>>>>>>>>> do
>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 512
>>>>>>>>>>> tx-usecs 128
>>>>>>>>>>> ethtool -K $i gro on
>>>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>>>
>>>>>>>>>>> done
>>>>>>>>>>>
>>>>>>>>>>> Server is leaking about 4-6MB per each 10 seconds
>>>>>>>>>>> MEMLEAK:
>>>>>>>>>>> 5 MB/10sec
>>>>>>>>>>> 6 MB/10sec
>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Other settings TSO/GRO off
>>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>>>> for i in $ifc
>>>>>>>>>>> do
>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 512
>>>>>>>>>>> tx-usecs 128
>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>>>
>>>>>>>>>>> done
>>>>>>>>>>>
>>>>>>>>>>> Same leak about 5MB per 10 seconds
>>>>>>>>>>> MEMLEAK:
>>>>>>>>>>> 5 MB/10sec
>>>>>>>>>>> 5 MB/10sec
>>>>>>>>>>> 5 MB/10sec
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Other settings rx-usecs change from 512 to 1024:
>>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>>>> for i in $ifc
>>>>>>>>>>> do
>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
>>>>>>>>>>> 1024 tx-usecs 128
>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>>>
>>>>>>>>>>> done
>>>>>>>>>>>
>>>>>>>>>>> MEMLEAK:
>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>> 3 MB/10sec
>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So memleak have something to do with rx-usecs (less interrupts but
>>>>>>>>>>> bigger latency for traffic)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> But also enabling TSO/GRO making leak about 1MB bigger for each 10
>>>>>>>>>>> seconds
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> So far best config is:
>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>>>>>>>> enp3s0f3'
>>>>>>>>>> for i in $ifc
>>>>>>>>>> do
>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 64
>>>>>>>>>> tx-usecs 512
>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>>
>>>>>>>>>> done
>>>>>>>>>>
>>>>>>>>>> MEMLEAK - about 2MB/10secs
>>>>>>>>>> 2 MB/10sec
>>>>>>>>>> 2 MB/10sec
>>>>>>>>>> 2 MB/10sec
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> With - rx-usecs set to 256 (about 7-9MB/10secs memleak)
>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>>>>>>>> enp3s0f3'
>>>>>>>>>> for i in $ifc
>>>>>>>>>> do
>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 256
>>>>>>>>>> tx-usecs 512
>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>>
>>>>>>>>>> done
>>>>>>>>>>
>>>>>>>>>> MEMLEAK:
>>>>>>>>>> 7 MB/10sec
>>>>>>>>>> 7 MB/10sec
>>>>>>>>>> 8 MB/10sec
>>>>>>>>>> 9 MB/10sec
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> And even less memleak with rx-usecs set to 32
>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>>>>>>> enp3s0f3'
>>>>>>>>> for i in $ifc
>>>>>>>>> do
>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 32
>>>>>>>>> tx-usecs 512
>>>>>>>>> ethtool -K $i gro off
>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>
>>>>>>>>> done
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> MEMLEAK - about 0-2MB for each 10 seconds
>>>>>>>>> 0 MB/10sec
>>>>>>>>> 1 MB/10sec
>>>>>>>>> 0 MB/10sec
>>>>>>>>> 2 MB/10sec
>>>>>>>>> 1 MB/10sec
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> So best settings - to have as less leak as possible for now (rx-usecs
>>>>>>>> set to 16):
>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>>>>>> enp3s0f3'
>>>>>>>> for i in $ifc
>>>>>>>> do
>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 16
>>>>>>>> tx-usecs 768
>>>>>>>> ethtool -K $i gro on
>>>>>>>> ethtool -K $i tso on
>>>>>>>>
>>>>>>>> done
>>>>>>>>
>>>>>>>>
>>>>>>>> MEMLEAK: (0-1MB/10seconds)
>>>>>>>> 0 MB/10sec
>>>>>>>> 0 MB/10sec
>>>>>>>> 0 MB/10sec
>>>>>>>> 1 MB/10sec
>>>>>>>> 1 MB/10sec
>>>>>>>> -1 MB/10sec
>>>>>>>> 1 MB/10sec
>>>>>>>> 1 MB/10sec
>>>>>>>> 0 MB/10sec
>>>>>>>>
>>>>>>>> (there are some memory recycles - so this is good :) )
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Compared to(rx-usecs 512):
>>>>>>>>
>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>>>>>> enp3s0f3'
>>>>>>>> for i in $ifc
>>>>>>>> do
>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 512
>>>>>>>> tx-usecs 128
>>>>>>>> ethtool -K $i gro on
>>>>>>>> ethtool -K $i tso on
>>>>>>>>
>>>>>>>> done
>>>>>>>>
>>>>>>>> Server is leaking about 4-6MB per each 10 seconds
>>>>>>>> MEMLEAK:
>>>>>>>> 5 MB/10sec
>>>>>>>> 6 MB/10sec
>>>>>>>> 4 MB/10sec
>>>>>>>> 4 MB/10sec
>>>>>>>>
>>>>>>>>
>>>>>>> And graph where all changes for rx-usecs was done over some time:
>>>>>>> https://ibb.co/nrRfbR
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Cant eliminate the problem with settings - memleak is bigger or less
>>>>>> visible with rx-usecs set to low values - but then have 100% cpu load - cant
>>>>>> have rx-usecs set to 16
>>>>>>
>>>>>> Cant find also other host with same cards or that are using i40e driver
>>>>>> for tests with bisecting
>>>>>> So will just replace to mellanox :)
>>>>>>
>>>>>>
>>>>> Also after fresh reboot with i40e
>>>>> startup settings:
>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>>> enp3s0f3'
>>>>> for i in $ifc
>>>>> do
>>>>> ip link set up dev $i
>>>>> ethtool -A $i autoneg off rx off tx off
>>>>> ethtool -G $i rx 2048 tx 2048
>>>>> ip link set $i txqueuelen 1000
>>>>> #ethtool -C $i rx-usecs 256
>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17
>>>>> tx-usecs 125
>>>>> ethtool -L $i combined 6
>>>>> #ethtool -N $i rx-flow-hash udp4 sdfn
>>>>> #ethtool -K $i ntuple on
>>>>> #ethtool -K $i gro off
>>>>> #ethtool -K $i tso off
>>>>> done
>>>>>
>>>>>
>>>>> After issuing:
>>>>>
>>>>> ethtool -K enp2s0f0 gro on tso on
>>>>>
>>>>> dmesg shows
>>>>> [35764.338259] i40e 0000:02:00.0: PF reset failed, -15
>>>>>
>>>>>
>>>>> and no traffic on the card :)
>>>>>
>>>>>
>>>> Also checked now
>>>> bigger rx ring
>>>> ethtool -G $i rx 2048 tx 2048
>>>>
>>>>
>>>> Bigger memleag :)
>>>>
>>>>
>>>>
>>> ok need to change cards now to ixgbe .... no reply no help for i40e so
>>> ....
>>>
>>> maybee someone else with i40e will gather more data i have only this host
>>> soo far - will try to install this cards to other hosts after change but
>>> alll this movement will takes about 2 maybee 3 months - nobody from my team
>>> want to but now cards that supports i40e cause of this bug soo this is hard
>>> now to debug - i need to change also all cards now >10G to mellanox that
>>> have no such bug ... sorry :)
>>>
>>>
>> Last tests from my side:)
>> settings
>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>> enp3s0f3'
>> for i in $ifc
>> do
>> ip link set up dev $i
>> ethtool -A $i autoneg off rx off tx off
>> ethtool -G $i rx 2048 tx 2048
>> ip link set $i txqueuelen 1000
>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17 tx-usecs
>> 125
>> ethtool -L $i combined 6
>> ethtool -K $i ntuple on
>> ethtool -K $i gro on
>> ethtool -K $i tso on
>> done
>>
>> MEMLEAK 1-2MB/10secs
>> 1 MB/10sec
>> 2 MB/10sec
>> 1 MB/10sec
>> 2 MB/10sec
>> 2 MB/10sec
>> 2 MB/10sec
>> 1 MB/10sec
>> 2 MB/10sec
>> 2 MB/10sec
>> 2 MB/10sec
>> 1 MB/10sec
>> 2 MB/10sec
>> 1 MB/10sec
>> 1 MB/10sec
>> 0 MB/10sec
>> 2 MB/10sec
>> 2 MB/10sec
>> 0 MB/10sec
>> 2 MB/10sec
>> 5 MB/10sec
>>
>> Change rx-usecs 16 tx usecs 16
>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>> enp3s0f3'
>> for i in $ifc
>> do
>> ip link set up dev $i
>> ethtool -A $i autoneg off rx off tx off
>> ethtool -G $i rx 2048 tx 2048
>> ip link set $i txqueuelen 1000
>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 16 tx-usecs
>> 16
>> ethtool -L $i combined 6
>> ethtool -K $i ntuple on
>> ethtool -K $i gro on
>> ethtool -K $i tso on
>> done
>>
>> MEMLEAK: 0-2MB/s with some recycles
>> 0 MB/10sec
>> 0 MB/10sec
>> 0 MB/10sec
>> 0 MB/10sec
>> 0 MB/10sec
>> 0 MB/10sec
>> 1 MB/10sec
>> 0 MB/10sec
>> 2 MB/10sec
>> 0 MB/10sec
>> 2 MB/10sec
>> -1 MB/10sec
>> 0 MB/10sec
>> 2 MB/10sec
>> 0 MB/10sec
>> 2 MB/10sec
>> -1 MB/10sec
>> 1 MB/10sec
> This data doesn't tell me much of anything and isn't what I asked for.
> I don't see how the interrupt throttling rate would be associated with
> your memory leak other than possibly rate limiting it by rate limiting
> the traffic itself. Is there something that gave you the impression
> that interrupt rate was somehow involved?
more interrupts more leak
>
> When we last talked I had asked if you could do a git bisect to find
> the memory leak and you said you would look into it. The most useful
> way to solve this would be to do a git bisect between your current
> kernel and the 4.11 kernel to find the point at which this started. If
> we can do that then fixing this becomes much simpler as we just have
> to fix the patch that introduced the issue.
>
> Also, I don't know it is you are using to determine that there is a
> memory leak. What tool is it you are using to do the tracking? Is
> there any specific form of traffic that is causing the leak? If you
> can't perform the bisection, any information you could provide that
> would allow me to do it would also be useful.
simple script
mem1=`free -m | grep Mem: | awk '{print $3}'`
sleep 10
mem2=`free -m | grep Mem: | awk '{print $3}'`
num=$((mem2 - mem1))
echo $num " MB/10sec"
There is nothing more that gets mem
there is only routed traffic from interface A to B
nothings takes mem
And memleaks only anchge when i change the rx/tx usecs for card
What You need more ?
imagine this is not my only prblem but many - i just want to help i
changed cards to i40e based only cause somebody rises a bug - and i want
to use i40e in feature - dont need them now - but maybee it is good to
help ppl to solwe some problems now if i can - before i will use this
cards ?
I try to use i40e before but there was bug covered by bug - and nobody
from e1000.sf can help me they just reply after year and closing tickets
with info about no activity but they have info in reported bugs ...
soooo what is this ? support center ? for me no .
If i want to help -= after a year response will be something like -
"dont care now" - cause i'v used other hw or sme hacks to repair problem
that should be sloved by intel
>
> Thanks.
>
> - Alex
>
^ permalink raw reply
* Re: Linux 4.12+ memory leak on router with i40e NICs
From: Paweł Staszewski @ 2017-10-18 23:56 UTC (permalink / raw)
To: Alexander Duyck
Cc: Pavlos Parissis, Anders K. Pedersen | Cohaesio,
netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org,
alexander.h.duyck@intel.com
In-Reply-To: <d4d1c562-e976-1894-e36c-709fd319dfb2@itcare.pl>
W dniu 2017-10-19 o 01:51, Paweł Staszewski pisze:
>
>
> W dniu 2017-10-19 o 01:37, Alexander Duyck pisze:
>> On Wed, Oct 18, 2017 at 4:22 PM, Paweł Staszewski
>> <pstaszewski@itcare.pl> wrote:
>>>
>>> W dniu 2017-10-19 o 00:58, Paweł Staszewski pisze:
>>>
>>>>
>>>> W dniu 2017-10-19 o 00:50, Paweł Staszewski pisze:
>>>>>
>>>>>
>>>>> W dniu 2017-10-19 o 00:20, Paweł Staszewski pisze:
>>>>>>
>>>>>>
>>>>>> W dniu 2017-10-18 o 17:44, Paweł Staszewski pisze:
>>>>>>>
>>>>>>>
>>>>>>> W dniu 2017-10-17 o 16:08, Paweł Staszewski pisze:
>>>>>>>>
>>>>>>>>
>>>>>>>> W dniu 2017-10-17 o 13:52, Paweł Staszewski pisze:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> W dniu 2017-10-17 o 13:05, Paweł Staszewski pisze:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> W dniu 2017-10-17 o 12:59, Paweł Staszewski pisze:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> W dniu 2017-10-17 o 12:51, Paweł Staszewski pisze:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> W dniu 2017-10-17 o 12:20, Paweł Staszewski pisze:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> W dniu 2017-10-17 o 11:48, Paweł Staszewski pisze:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> W dniu 2017-10-17 o 02:44, Paweł Staszewski pisze:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> W dniu 2017-10-17 o 01:56, Alexander Duyck pisze:
>>>>>>>>>>>>>>>> On Mon, Oct 16, 2017 at 4:34 PM, Paweł Staszewski
>>>>>>>>>>>>>>>> <pstaszewski@itcare.pl> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> W dniu 2017-10-16 o 18:26, Paweł Staszewski pisze:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> W dniu 2017-10-16 o 13:20, Pavlos Parissis pisze:
>>>>>>>>>>>>>>>>>>> On 15/10/2017 02:58 πμ, Alexander Duyck wrote:
>>>>>>>>>>>>>>>>>>>> Hi Pawel,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> To clarify is that Dave Miller's tree or Linus's
>>>>>>>>>>>>>>>>>>>> that you
>>>>>>>>>>>>>>>>>>>> are talking
>>>>>>>>>>>>>>>>>>>> about? If it is Dave's tree how long ago was it you
>>>>>>>>>>>>>>>>>>>> pulled
>>>>>>>>>>>>>>>>>>>> it since I
>>>>>>>>>>>>>>>>>>>> think the fix was just pushed by Jeff Kirsher a few
>>>>>>>>>>>>>>>>>>>> days
>>>>>>>>>>>>>>>>>>>> ago.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The issue should be fixed in the following commit:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/drivers/net/ethernet/intel/i40e/i40e_txrx.c?id=2b9478ffc550f17c6cd8c69057234e91150f5972
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Do you know when it is going to be available on
>>>>>>>>>>>>>>>>>>> net-next
>>>>>>>>>>>>>>>>>>> and linux-stable
>>>>>>>>>>>>>>>>>>> repos?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>> Pavlos
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I will make some tests today night with "net" git
>>>>>>>>>>>>>>>>>> tree where
>>>>>>>>>>>>>>>>>> this patch is
>>>>>>>>>>>>>>>>>> included.
>>>>>>>>>>>>>>>>>> Starting from 0:00 CET
>>>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Upgraded and looks like problem is not solved with
>>>>>>>>>>>>>>>>> that patch
>>>>>>>>>>>>>>>>> Currently running system with
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Still about 0.5GB of memory is leaking somewhere
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also can confirm that the latest kernel where memory
>>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>>> leaking (with
>>>>>>>>>>>>>>>>> use i40e driver intel 710 cards) is 4.11.12
>>>>>>>>>>>>>>>>> With kernel 4.11.12 - after hour no change in memory
>>>>>>>>>>>>>>>>> usage.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> also checked that with ixgbe instead of i40e with same
>>>>>>>>>>>>>>>>> net.git kernel there
>>>>>>>>>>>>>>>>> is no memleak - after hour same memory usage - so for
>>>>>>>>>>>>>>>>> 100%
>>>>>>>>>>>>>>>>> this is i40e
>>>>>>>>>>>>>>>>> driver problem.
>>>>>>>>>>>>>>>> So how long was the run to get the .5GB of memory leaking?
>>>>>>>>>>>>>>> 1 hour
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also is there any chance of you being able to bisect to
>>>>>>>>>>>>>>>> determine
>>>>>>>>>>>>>>>> where the memory leak was introduced since as you
>>>>>>>>>>>>>>>> pointed out
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> didn't exist in 4.11.12 so odds are it was introduced
>>>>>>>>>>>>>>>> somewhere
>>>>>>>>>>>>>>>> between 4.11 and the latest kernel release.
>>>>>>>>>>>>>>> Can be hard cause currently need to back to 4.11.12 -
>>>>>>>>>>>>>>> this is
>>>>>>>>>>>>>>> production host/router
>>>>>>>>>>>>>>> Will try to find some free/test router for tests/bicects
>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>> i40e driver (intel 710 cards)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Alex
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also forgoto to add errors for i40e when driver initialize:
>>>>>>>>>>>>>> [ 15.760569] i40e 0000:02:00.1: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>> [ 16.365587] i40e 0000:03:00.3: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>> [ 16.367686] i40e 0000:02:00.2: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>> [ 16.368816] i40e 0000:03:00.0: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>> [ 16.369877] i40e 0000:03:00.2: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>> [ 16.370941] i40e 0000:02:00.3: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>> [ 16.372005] i40e 0000:02:00.0: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>> [ 16.373029] i40e 0000:03:00.1: Error I40E_AQ_RC_ENOSPC
>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> some params that are set for this nic's
>>>>>>>>>>>>>> ip link set up dev $i
>>>>>>>>>>>>>> ethtool -A $i autoneg off rx off tx off
>>>>>>>>>>>>>> ethtool -G $i rx 1024 tx 2048
>>>>>>>>>>>>>> ip link set $i txqueuelen 1000
>>>>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>>>>>> rx-usecs
>>>>>>>>>>>>>> 512 tx-usecs 128
>>>>>>>>>>>>>> ethtool -L $i combined 6
>>>>>>>>>>>>>> #ethtool -N $i rx-flow-hash udp4 sdfn
>>>>>>>>>>>>>> ethtool -K $i ntuple on
>>>>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Also after TSO/GRO on there is memory usage change - and
>>>>>>>>>>>>> leaking
>>>>>>>>>>>>> faster
>>>>>>>>>>>>> Below image from memory usage before change with TSO/GRO
>>>>>>>>>>>>> OFF and
>>>>>>>>>>>>> after enabling TSO/GRO
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://ibb.co/dTqBY6
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Pawel
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> With settings like this:
>>>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>>>>> for i in $ifc
>>>>>>>>>>>> do
>>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>>>> rx-usecs 512
>>>>>>>>>>>> tx-usecs 128
>>>>>>>>>>>> ethtool -K $i gro on
>>>>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>>>>
>>>>>>>>>>>> done
>>>>>>>>>>>>
>>>>>>>>>>>> Server is leaking about 4-6MB per each 10 seconds
>>>>>>>>>>>> MEMLEAK:
>>>>>>>>>>>> 5 MB/10sec
>>>>>>>>>>>> 6 MB/10sec
>>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Other settings TSO/GRO off
>>>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>>>>> for i in $ifc
>>>>>>>>>>>> do
>>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>>>> rx-usecs 512
>>>>>>>>>>>> tx-usecs 128
>>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>>>>
>>>>>>>>>>>> done
>>>>>>>>>>>>
>>>>>>>>>>>> Same leak about 5MB per 10 seconds
>>>>>>>>>>>> MEMLEAK:
>>>>>>>>>>>> 5 MB/10sec
>>>>>>>>>>>> 5 MB/10sec
>>>>>>>>>>>> 5 MB/10sec
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Other settings rx-usecs change from 512 to 1024:
>>>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>>>>> for i in $ifc
>>>>>>>>>>>> do
>>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>>>> rx-usecs
>>>>>>>>>>>> 1024 tx-usecs 128
>>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>>>>
>>>>>>>>>>>> done
>>>>>>>>>>>>
>>>>>>>>>>>> MEMLEAK:
>>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>>> 3 MB/10sec
>>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> So memleak have something to do with rx-usecs (less
>>>>>>>>>>>> interrupts but
>>>>>>>>>>>> bigger latency for traffic)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> But also enabling TSO/GRO making leak about 1MB bigger for
>>>>>>>>>>>> each 10
>>>>>>>>>>>> seconds
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> So far best config is:
>>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>>> enp3s0f2
>>>>>>>>>>> enp3s0f3'
>>>>>>>>>>> for i in $ifc
>>>>>>>>>>> do
>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>>> rx-usecs 64
>>>>>>>>>>> tx-usecs 512
>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>>>
>>>>>>>>>>> done
>>>>>>>>>>>
>>>>>>>>>>> MEMLEAK - about 2MB/10secs
>>>>>>>>>>> 2 MB/10sec
>>>>>>>>>>> 2 MB/10sec
>>>>>>>>>>> 2 MB/10sec
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> With - rx-usecs set to 256 (about 7-9MB/10secs memleak)
>>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>>> enp3s0f2
>>>>>>>>>>> enp3s0f3'
>>>>>>>>>>> for i in $ifc
>>>>>>>>>>> do
>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>>> rx-usecs 256
>>>>>>>>>>> tx-usecs 512
>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>>>
>>>>>>>>>>> done
>>>>>>>>>>>
>>>>>>>>>>> MEMLEAK:
>>>>>>>>>>> 7 MB/10sec
>>>>>>>>>>> 7 MB/10sec
>>>>>>>>>>> 8 MB/10sec
>>>>>>>>>>> 9 MB/10sec
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> And even less memleak with rx-usecs set to 32
>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>> enp3s0f2
>>>>>>>>>> enp3s0f3'
>>>>>>>>>> for i in $ifc
>>>>>>>>>> do
>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>> rx-usecs 32
>>>>>>>>>> tx-usecs 512
>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>>
>>>>>>>>>> done
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> MEMLEAK - about 0-2MB for each 10 seconds
>>>>>>>>>> 0 MB/10sec
>>>>>>>>>> 1 MB/10sec
>>>>>>>>>> 0 MB/10sec
>>>>>>>>>> 2 MB/10sec
>>>>>>>>>> 1 MB/10sec
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So best settings - to have as less leak as possible for now
>>>>>>>>> (rx-usecs
>>>>>>>>> set to 16):
>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>> enp3s0f2
>>>>>>>>> enp3s0f3'
>>>>>>>>> for i in $ifc
>>>>>>>>> do
>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>> rx-usecs 16
>>>>>>>>> tx-usecs 768
>>>>>>>>> ethtool -K $i gro on
>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>
>>>>>>>>> done
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> MEMLEAK: (0-1MB/10seconds)
>>>>>>>>> 0 MB/10sec
>>>>>>>>> 0 MB/10sec
>>>>>>>>> 0 MB/10sec
>>>>>>>>> 1 MB/10sec
>>>>>>>>> 1 MB/10sec
>>>>>>>>> -1 MB/10sec
>>>>>>>>> 1 MB/10sec
>>>>>>>>> 1 MB/10sec
>>>>>>>>> 0 MB/10sec
>>>>>>>>>
>>>>>>>>> (there are some memory recycles - so this is good :) )
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Compared to(rx-usecs 512):
>>>>>>>>>
>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>> enp3s0f2
>>>>>>>>> enp3s0f3'
>>>>>>>>> for i in $ifc
>>>>>>>>> do
>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>> rx-usecs 512
>>>>>>>>> tx-usecs 128
>>>>>>>>> ethtool -K $i gro on
>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>
>>>>>>>>> done
>>>>>>>>>
>>>>>>>>> Server is leaking about 4-6MB per each 10 seconds
>>>>>>>>> MEMLEAK:
>>>>>>>>> 5 MB/10sec
>>>>>>>>> 6 MB/10sec
>>>>>>>>> 4 MB/10sec
>>>>>>>>> 4 MB/10sec
>>>>>>>>>
>>>>>>>>>
>>>>>>>> And graph where all changes for rx-usecs was done over some time:
>>>>>>>> https://ibb.co/nrRfbR
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Cant eliminate the problem with settings - memleak is bigger or
>>>>>>> less
>>>>>>> visible with rx-usecs set to low values - but then have 100% cpu
>>>>>>> load - cant
>>>>>>> have rx-usecs set to 16
>>>>>>>
>>>>>>> Cant find also other host with same cards or that are using i40e
>>>>>>> driver
>>>>>>> for tests with bisecting
>>>>>>> So will just replace to mellanox :)
>>>>>>>
>>>>>>>
>>>>>> Also after fresh reboot with i40e
>>>>>> startup settings:
>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>>>> enp3s0f3'
>>>>>> for i in $ifc
>>>>>> do
>>>>>> ip link set up dev $i
>>>>>> ethtool -A $i autoneg off rx off tx off
>>>>>> ethtool -G $i rx 2048 tx 2048
>>>>>> ip link set $i txqueuelen 1000
>>>>>> #ethtool -C $i rx-usecs 256
>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17
>>>>>> tx-usecs 125
>>>>>> ethtool -L $i combined 6
>>>>>> #ethtool -N $i rx-flow-hash udp4 sdfn
>>>>>> #ethtool -K $i ntuple on
>>>>>> #ethtool -K $i gro off
>>>>>> #ethtool -K $i tso off
>>>>>> done
>>>>>>
>>>>>>
>>>>>> After issuing:
>>>>>>
>>>>>> ethtool -K enp2s0f0 gro on tso on
>>>>>>
>>>>>> dmesg shows
>>>>>> [35764.338259] i40e 0000:02:00.0: PF reset failed, -15
>>>>>>
>>>>>>
>>>>>> and no traffic on the card :)
>>>>>>
>>>>>>
>>>>> Also checked now
>>>>> bigger rx ring
>>>>> ethtool -G $i rx 2048 tx 2048
>>>>>
>>>>>
>>>>> Bigger memleag :)
>>>>>
>>>>>
>>>>>
>>>> ok need to change cards now to ixgbe .... no reply no help for i40e so
>>>> ....
>>>>
>>>> maybee someone else with i40e will gather more data i have only
>>>> this host
>>>> soo far - will try to install this cards to other hosts after
>>>> change but
>>>> alll this movement will takes about 2 maybee 3 months - nobody from
>>>> my team
>>>> want to but now cards that supports i40e cause of this bug soo this
>>>> is hard
>>>> now to debug - i need to change also all cards now >10G to mellanox
>>>> that
>>>> have no such bug ... sorry :)
>>>>
>>>>
>>> Last tests from my side:)
>>> settings
>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>> enp3s0f3'
>>> for i in $ifc
>>> do
>>> ip link set up dev $i
>>> ethtool -A $i autoneg off rx off tx off
>>> ethtool -G $i rx 2048 tx 2048
>>> ip link set $i txqueuelen 1000
>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17
>>> tx-usecs
>>> 125
>>> ethtool -L $i combined 6
>>> ethtool -K $i ntuple on
>>> ethtool -K $i gro on
>>> ethtool -K $i tso on
>>> done
>>>
>>> MEMLEAK 1-2MB/10secs
>>> 1 MB/10sec
>>> 2 MB/10sec
>>> 1 MB/10sec
>>> 2 MB/10sec
>>> 2 MB/10sec
>>> 2 MB/10sec
>>> 1 MB/10sec
>>> 2 MB/10sec
>>> 2 MB/10sec
>>> 2 MB/10sec
>>> 1 MB/10sec
>>> 2 MB/10sec
>>> 1 MB/10sec
>>> 1 MB/10sec
>>> 0 MB/10sec
>>> 2 MB/10sec
>>> 2 MB/10sec
>>> 0 MB/10sec
>>> 2 MB/10sec
>>> 5 MB/10sec
>>>
>>> Change rx-usecs 16 tx usecs 16
>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>> enp3s0f3'
>>> for i in $ifc
>>> do
>>> ip link set up dev $i
>>> ethtool -A $i autoneg off rx off tx off
>>> ethtool -G $i rx 2048 tx 2048
>>> ip link set $i txqueuelen 1000
>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 16
>>> tx-usecs
>>> 16
>>> ethtool -L $i combined 6
>>> ethtool -K $i ntuple on
>>> ethtool -K $i gro on
>>> ethtool -K $i tso on
>>> done
>>>
>>> MEMLEAK: 0-2MB/s with some recycles
>>> 0 MB/10sec
>>> 0 MB/10sec
>>> 0 MB/10sec
>>> 0 MB/10sec
>>> 0 MB/10sec
>>> 0 MB/10sec
>>> 1 MB/10sec
>>> 0 MB/10sec
>>> 2 MB/10sec
>>> 0 MB/10sec
>>> 2 MB/10sec
>>> -1 MB/10sec
>>> 0 MB/10sec
>>> 2 MB/10sec
>>> 0 MB/10sec
>>> 2 MB/10sec
>>> -1 MB/10sec
>>> 1 MB/10sec
>> This data doesn't tell me much of anything and isn't what I asked for.
>> I don't see how the interrupt throttling rate would be associated with
>> your memory leak other than possibly rate limiting it by rate limiting
>> the traffic itself. Is there something that gave you the impression
>> that interrupt rate was somehow involved?
> more interrupts more leak
>
>>
>> When we last talked I had asked if you could do a git bisect to find
>> the memory leak and you said you would look into it. The most useful
>> way to solve this would be to do a git bisect between your current
>> kernel and the 4.11 kernel to find the point at which this started. If
>> we can do that then fixing this becomes much simpler as we just have
>> to fix the patch that introduced the issue.
>>
>> Also, I don't know it is you are using to determine that there is a
>> memory leak. What tool is it you are using to do the tracking? Is
>> there any specific form of traffic that is causing the leak? If you
>> can't perform the bisection, any information you could provide that
>> would allow me to do it would also be useful.
> simple script
>
> mem1=`free -m | grep Mem: | awk '{print $3}'`
> sleep 10
> mem2=`free -m | grep Mem: | awk '{print $3}'`
>
> num=$((mem2 - mem1))
> echo $num " MB/10sec"
>
>
> There is nothing more that gets mem
> there is only routed traffic from interface A to B
> nothings takes mem
> And memleaks only anchge when i change the rx/tx usecs for card
>
> What You need more ?
>
> imagine this is not my only prblem but many - i just want to help i
> changed cards to i40e based only cause somebody rises a bug - and i
> want to use i40e in feature - dont need them now - but maybee it is
> good to help ppl to solwe some problems now if i can - before i will
> use this cards ?
> I try to use i40e before but there was bug covered by bug - and nobody
> from e1000.sf can help me they just reply after year and closing
> tickets with info about no activity but they have info in reported
> bugs ... soooo what is this ? support center ? for me no .
> If i want to help -= after a year response will be something like -
> "dont care now" - cause i'v used other hw or sme hacks to repair
> problem that should be sloved by intel
>
>>
>> Thanks.
>>
>> - Alex
>>
>
>
What i can say more
is that
if:
adaptive-rx off
adaptive-tx off
rx-usecs 10
tx-usecs 10
There is almost no memleak
but i dont know if this is problem rx-usecx=tx-usecs - then no memleak
or just lower numbers for rx/tx-usecs - are doing this
But if You see my graphs You will see that less rx-usecs = less memleak
^ permalink raw reply
* Re: Linux 4.12+ memory leak on router with i40e NICs
From: Paweł Staszewski @ 2017-10-18 23:59 UTC (permalink / raw)
To: Alexander Duyck
Cc: Pavlos Parissis, Anders K. Pedersen | Cohaesio,
netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org,
alexander.h.duyck@intel.com
In-Reply-To: <e7dc9b20-c137-6095-1106-61d394142fec@itcare.pl>
W dniu 2017-10-19 o 01:56, Paweł Staszewski pisze:
>
>
> W dniu 2017-10-19 o 01:51, Paweł Staszewski pisze:
>>
>>
>> W dniu 2017-10-19 o 01:37, Alexander Duyck pisze:
>>> On Wed, Oct 18, 2017 at 4:22 PM, Paweł Staszewski
>>> <pstaszewski@itcare.pl> wrote:
>>>>
>>>> W dniu 2017-10-19 o 00:58, Paweł Staszewski pisze:
>>>>
>>>>>
>>>>> W dniu 2017-10-19 o 00:50, Paweł Staszewski pisze:
>>>>>>
>>>>>>
>>>>>> W dniu 2017-10-19 o 00:20, Paweł Staszewski pisze:
>>>>>>>
>>>>>>>
>>>>>>> W dniu 2017-10-18 o 17:44, Paweł Staszewski pisze:
>>>>>>>>
>>>>>>>>
>>>>>>>> W dniu 2017-10-17 o 16:08, Paweł Staszewski pisze:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> W dniu 2017-10-17 o 13:52, Paweł Staszewski pisze:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> W dniu 2017-10-17 o 13:05, Paweł Staszewski pisze:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> W dniu 2017-10-17 o 12:59, Paweł Staszewski pisze:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> W dniu 2017-10-17 o 12:51, Paweł Staszewski pisze:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> W dniu 2017-10-17 o 12:20, Paweł Staszewski pisze:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> W dniu 2017-10-17 o 11:48, Paweł Staszewski pisze:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> W dniu 2017-10-17 o 02:44, Paweł Staszewski pisze:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> W dniu 2017-10-17 o 01:56, Alexander Duyck pisze:
>>>>>>>>>>>>>>>>> On Mon, Oct 16, 2017 at 4:34 PM, Paweł Staszewski
>>>>>>>>>>>>>>>>> <pstaszewski@itcare.pl> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> W dniu 2017-10-16 o 18:26, Paweł Staszewski pisze:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> W dniu 2017-10-16 o 13:20, Pavlos Parissis pisze:
>>>>>>>>>>>>>>>>>>>> On 15/10/2017 02:58 πμ, Alexander Duyck wrote:
>>>>>>>>>>>>>>>>>>>>> Hi Pawel,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> To clarify is that Dave Miller's tree or Linus's
>>>>>>>>>>>>>>>>>>>>> that you
>>>>>>>>>>>>>>>>>>>>> are talking
>>>>>>>>>>>>>>>>>>>>> about? If it is Dave's tree how long ago was it
>>>>>>>>>>>>>>>>>>>>> you pulled
>>>>>>>>>>>>>>>>>>>>> it since I
>>>>>>>>>>>>>>>>>>>>> think the fix was just pushed by Jeff Kirsher a
>>>>>>>>>>>>>>>>>>>>> few days
>>>>>>>>>>>>>>>>>>>>> ago.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The issue should be fixed in the following commit:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/drivers/net/ethernet/intel/i40e/i40e_txrx.c?id=2b9478ffc550f17c6cd8c69057234e91150f5972
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Do you know when it is going to be available on
>>>>>>>>>>>>>>>>>>>> net-next
>>>>>>>>>>>>>>>>>>>> and linux-stable
>>>>>>>>>>>>>>>>>>>> repos?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>> Pavlos
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I will make some tests today night with "net" git
>>>>>>>>>>>>>>>>>>> tree where
>>>>>>>>>>>>>>>>>>> this patch is
>>>>>>>>>>>>>>>>>>> included.
>>>>>>>>>>>>>>>>>>> Starting from 0:00 CET
>>>>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Upgraded and looks like problem is not solved with
>>>>>>>>>>>>>>>>>> that patch
>>>>>>>>>>>>>>>>>> Currently running system with
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Still about 0.5GB of memory is leaking somewhere
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Also can confirm that the latest kernel where memory
>>>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>>>> leaking (with
>>>>>>>>>>>>>>>>>> use i40e driver intel 710 cards) is 4.11.12
>>>>>>>>>>>>>>>>>> With kernel 4.11.12 - after hour no change in memory
>>>>>>>>>>>>>>>>>> usage.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> also checked that with ixgbe instead of i40e with same
>>>>>>>>>>>>>>>>>> net.git kernel there
>>>>>>>>>>>>>>>>>> is no memleak - after hour same memory usage - so for
>>>>>>>>>>>>>>>>>> 100%
>>>>>>>>>>>>>>>>>> this is i40e
>>>>>>>>>>>>>>>>>> driver problem.
>>>>>>>>>>>>>>>>> So how long was the run to get the .5GB of memory
>>>>>>>>>>>>>>>>> leaking?
>>>>>>>>>>>>>>>> 1 hour
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also is there any chance of you being able to bisect to
>>>>>>>>>>>>>>>>> determine
>>>>>>>>>>>>>>>>> where the memory leak was introduced since as you
>>>>>>>>>>>>>>>>> pointed out
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> didn't exist in 4.11.12 so odds are it was introduced
>>>>>>>>>>>>>>>>> somewhere
>>>>>>>>>>>>>>>>> between 4.11 and the latest kernel release.
>>>>>>>>>>>>>>>> Can be hard cause currently need to back to 4.11.12 -
>>>>>>>>>>>>>>>> this is
>>>>>>>>>>>>>>>> production host/router
>>>>>>>>>>>>>>>> Will try to find some free/test router for
>>>>>>>>>>>>>>>> tests/bicects with
>>>>>>>>>>>>>>>> i40e driver (intel 710 cards)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Alex
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also forgoto to add errors for i40e when driver initialize:
>>>>>>>>>>>>>>> [ 15.760569] i40e 0000:02:00.1: Error
>>>>>>>>>>>>>>> I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>>> [ 16.365587] i40e 0000:03:00.3: Error
>>>>>>>>>>>>>>> I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>>> [ 16.367686] i40e 0000:02:00.2: Error
>>>>>>>>>>>>>>> I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>>> [ 16.368816] i40e 0000:03:00.0: Error
>>>>>>>>>>>>>>> I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>>> [ 16.369877] i40e 0000:03:00.2: Error
>>>>>>>>>>>>>>> I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>>> [ 16.370941] i40e 0000:02:00.3: Error
>>>>>>>>>>>>>>> I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>>> [ 16.372005] i40e 0000:02:00.0: Error
>>>>>>>>>>>>>>> I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>>> [ 16.373029] i40e 0000:03:00.1: Error
>>>>>>>>>>>>>>> I40E_AQ_RC_ENOSPC adding
>>>>>>>>>>>>>>> RX filters on PF, promiscuous mode forced on
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> some params that are set for this nic's
>>>>>>>>>>>>>>> ip link set up dev $i
>>>>>>>>>>>>>>> ethtool -A $i autoneg off rx off tx off
>>>>>>>>>>>>>>> ethtool -G $i rx 1024 tx 2048
>>>>>>>>>>>>>>> ip link set $i txqueuelen 1000
>>>>>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>>>>>>> rx-usecs
>>>>>>>>>>>>>>> 512 tx-usecs 128
>>>>>>>>>>>>>>> ethtool -L $i combined 6
>>>>>>>>>>>>>>> #ethtool -N $i rx-flow-hash udp4 sdfn
>>>>>>>>>>>>>>> ethtool -K $i ntuple on
>>>>>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also after TSO/GRO on there is memory usage change - and
>>>>>>>>>>>>>> leaking
>>>>>>>>>>>>>> faster
>>>>>>>>>>>>>> Below image from memory usage before change with TSO/GRO
>>>>>>>>>>>>>> OFF and
>>>>>>>>>>>>>> after enabling TSO/GRO
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://ibb.co/dTqBY6
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Pawel
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> With settings like this:
>>>>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>>>>>> for i in $ifc
>>>>>>>>>>>>> do
>>>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>>>>> rx-usecs 512
>>>>>>>>>>>>> tx-usecs 128
>>>>>>>>>>>>> ethtool -K $i gro on
>>>>>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>>>>>
>>>>>>>>>>>>> done
>>>>>>>>>>>>>
>>>>>>>>>>>>> Server is leaking about 4-6MB per each 10 seconds
>>>>>>>>>>>>> MEMLEAK:
>>>>>>>>>>>>> 5 MB/10sec
>>>>>>>>>>>>> 6 MB/10sec
>>>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Other settings TSO/GRO off
>>>>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>>>>>> for i in $ifc
>>>>>>>>>>>>> do
>>>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>>>>> rx-usecs 512
>>>>>>>>>>>>> tx-usecs 128
>>>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>>>>>
>>>>>>>>>>>>> done
>>>>>>>>>>>>>
>>>>>>>>>>>>> Same leak about 5MB per 10 seconds
>>>>>>>>>>>>> MEMLEAK:
>>>>>>>>>>>>> 5 MB/10sec
>>>>>>>>>>>>> 5 MB/10sec
>>>>>>>>>>>>> 5 MB/10sec
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Other settings rx-usecs change from 512 to 1024:
>>>>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>>>>> enp3s0f2 enp3s0f3'
>>>>>>>>>>>>> for i in $ifc
>>>>>>>>>>>>> do
>>>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>>>>> rx-usecs
>>>>>>>>>>>>> 1024 tx-usecs 128
>>>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>>>> ethtool -K $i tso off
>>>>>>>>>>>>>
>>>>>>>>>>>>> done
>>>>>>>>>>>>>
>>>>>>>>>>>>> MEMLEAK:
>>>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>>>> 3 MB/10sec
>>>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> So memleak have something to do with rx-usecs (less
>>>>>>>>>>>>> interrupts but
>>>>>>>>>>>>> bigger latency for traffic)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> But also enabling TSO/GRO making leak about 1MB bigger for
>>>>>>>>>>>>> each 10
>>>>>>>>>>>>> seconds
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> So far best config is:
>>>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>>>> enp3s0f2
>>>>>>>>>>>> enp3s0f3'
>>>>>>>>>>>> for i in $ifc
>>>>>>>>>>>> do
>>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>>>> rx-usecs 64
>>>>>>>>>>>> tx-usecs 512
>>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>>>>
>>>>>>>>>>>> done
>>>>>>>>>>>>
>>>>>>>>>>>> MEMLEAK - about 2MB/10secs
>>>>>>>>>>>> 2 MB/10sec
>>>>>>>>>>>> 2 MB/10sec
>>>>>>>>>>>> 2 MB/10sec
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> With - rx-usecs set to 256 (about 7-9MB/10secs memleak)
>>>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>>>> enp3s0f2
>>>>>>>>>>>> enp3s0f3'
>>>>>>>>>>>> for i in $ifc
>>>>>>>>>>>> do
>>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>>>> rx-usecs 256
>>>>>>>>>>>> tx-usecs 512
>>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>>>>
>>>>>>>>>>>> done
>>>>>>>>>>>>
>>>>>>>>>>>> MEMLEAK:
>>>>>>>>>>>> 7 MB/10sec
>>>>>>>>>>>> 7 MB/10sec
>>>>>>>>>>>> 8 MB/10sec
>>>>>>>>>>>> 9 MB/10sec
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> And even less memleak with rx-usecs set to 32
>>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>>> enp3s0f2
>>>>>>>>>>> enp3s0f3'
>>>>>>>>>>> for i in $ifc
>>>>>>>>>>> do
>>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>>> rx-usecs 32
>>>>>>>>>>> tx-usecs 512
>>>>>>>>>>> ethtool -K $i gro off
>>>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>>>
>>>>>>>>>>> done
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> MEMLEAK - about 0-2MB for each 10 seconds
>>>>>>>>>>> 0 MB/10sec
>>>>>>>>>>> 1 MB/10sec
>>>>>>>>>>> 0 MB/10sec
>>>>>>>>>>> 2 MB/10sec
>>>>>>>>>>> 1 MB/10sec
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> So best settings - to have as less leak as possible for now
>>>>>>>>>> (rx-usecs
>>>>>>>>>> set to 16):
>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>> enp3s0f2
>>>>>>>>>> enp3s0f3'
>>>>>>>>>> for i in $ifc
>>>>>>>>>> do
>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>> rx-usecs 16
>>>>>>>>>> tx-usecs 768
>>>>>>>>>> ethtool -K $i gro on
>>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>>
>>>>>>>>>> done
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> MEMLEAK: (0-1MB/10seconds)
>>>>>>>>>> 0 MB/10sec
>>>>>>>>>> 0 MB/10sec
>>>>>>>>>> 0 MB/10sec
>>>>>>>>>> 1 MB/10sec
>>>>>>>>>> 1 MB/10sec
>>>>>>>>>> -1 MB/10sec
>>>>>>>>>> 1 MB/10sec
>>>>>>>>>> 1 MB/10sec
>>>>>>>>>> 0 MB/10sec
>>>>>>>>>>
>>>>>>>>>> (there are some memory recycles - so this is good :) )
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Compared to(rx-usecs 512):
>>>>>>>>>>
>>>>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
>>>>>>>>>> enp3s0f2
>>>>>>>>>> enp3s0f3'
>>>>>>>>>> for i in $ifc
>>>>>>>>>> do
>>>>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off
>>>>>>>>>> rx-usecs 512
>>>>>>>>>> tx-usecs 128
>>>>>>>>>> ethtool -K $i gro on
>>>>>>>>>> ethtool -K $i tso on
>>>>>>>>>>
>>>>>>>>>> done
>>>>>>>>>>
>>>>>>>>>> Server is leaking about 4-6MB per each 10 seconds
>>>>>>>>>> MEMLEAK:
>>>>>>>>>> 5 MB/10sec
>>>>>>>>>> 6 MB/10sec
>>>>>>>>>> 4 MB/10sec
>>>>>>>>>> 4 MB/10sec
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> And graph where all changes for rx-usecs was done over some
>>>>>>>>> time:
>>>>>>>>> https://ibb.co/nrRfbR
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Cant eliminate the problem with settings - memleak is bigger or
>>>>>>>> less
>>>>>>>> visible with rx-usecs set to low values - but then have 100%
>>>>>>>> cpu load - cant
>>>>>>>> have rx-usecs set to 16
>>>>>>>>
>>>>>>>> Cant find also other host with same cards or that are using
>>>>>>>> i40e driver
>>>>>>>> for tests with bisecting
>>>>>>>> So will just replace to mellanox :)
>>>>>>>>
>>>>>>>>
>>>>>>> Also after fresh reboot with i40e
>>>>>>> startup settings:
>>>>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>>>>> enp3s0f3'
>>>>>>> for i in $ifc
>>>>>>> do
>>>>>>> ip link set up dev $i
>>>>>>> ethtool -A $i autoneg off rx off tx off
>>>>>>> ethtool -G $i rx 2048 tx 2048
>>>>>>> ip link set $i txqueuelen 1000
>>>>>>> #ethtool -C $i rx-usecs 256
>>>>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17
>>>>>>> tx-usecs 125
>>>>>>> ethtool -L $i combined 6
>>>>>>> #ethtool -N $i rx-flow-hash udp4 sdfn
>>>>>>> #ethtool -K $i ntuple on
>>>>>>> #ethtool -K $i gro off
>>>>>>> #ethtool -K $i tso off
>>>>>>> done
>>>>>>>
>>>>>>>
>>>>>>> After issuing:
>>>>>>>
>>>>>>> ethtool -K enp2s0f0 gro on tso on
>>>>>>>
>>>>>>> dmesg shows
>>>>>>> [35764.338259] i40e 0000:02:00.0: PF reset failed, -15
>>>>>>>
>>>>>>>
>>>>>>> and no traffic on the card :)
>>>>>>>
>>>>>>>
>>>>>> Also checked now
>>>>>> bigger rx ring
>>>>>> ethtool -G $i rx 2048 tx 2048
>>>>>>
>>>>>>
>>>>>> Bigger memleag :)
>>>>>>
>>>>>>
>>>>>>
>>>>> ok need to change cards now to ixgbe .... no reply no help for
>>>>> i40e so
>>>>> ....
>>>>>
>>>>> maybee someone else with i40e will gather more data i have only
>>>>> this host
>>>>> soo far - will try to install this cards to other hosts after
>>>>> change but
>>>>> alll this movement will takes about 2 maybee 3 months - nobody
>>>>> from my team
>>>>> want to but now cards that supports i40e cause of this bug soo
>>>>> this is hard
>>>>> now to debug - i need to change also all cards now >10G to
>>>>> mellanox that
>>>>> have no such bug ... sorry :)
>>>>>
>>>>>
>>>> Last tests from my side:)
>>>> settings
>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>> enp3s0f3'
>>>> for i in $ifc
>>>> do
>>>> ip link set up dev $i
>>>> ethtool -A $i autoneg off rx off tx off
>>>> ethtool -G $i rx 2048 tx 2048
>>>> ip link set $i txqueuelen 1000
>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17
>>>> tx-usecs
>>>> 125
>>>> ethtool -L $i combined 6
>>>> ethtool -K $i ntuple on
>>>> ethtool -K $i gro on
>>>> ethtool -K $i tso on
>>>> done
>>>>
>>>> MEMLEAK 1-2MB/10secs
>>>> 1 MB/10sec
>>>> 2 MB/10sec
>>>> 1 MB/10sec
>>>> 2 MB/10sec
>>>> 2 MB/10sec
>>>> 2 MB/10sec
>>>> 1 MB/10sec
>>>> 2 MB/10sec
>>>> 2 MB/10sec
>>>> 2 MB/10sec
>>>> 1 MB/10sec
>>>> 2 MB/10sec
>>>> 1 MB/10sec
>>>> 1 MB/10sec
>>>> 0 MB/10sec
>>>> 2 MB/10sec
>>>> 2 MB/10sec
>>>> 0 MB/10sec
>>>> 2 MB/10sec
>>>> 5 MB/10sec
>>>>
>>>> Change rx-usecs 16 tx usecs 16
>>>> ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
>>>> enp3s0f3'
>>>> for i in $ifc
>>>> do
>>>> ip link set up dev $i
>>>> ethtool -A $i autoneg off rx off tx off
>>>> ethtool -G $i rx 2048 tx 2048
>>>> ip link set $i txqueuelen 1000
>>>> ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 16
>>>> tx-usecs
>>>> 16
>>>> ethtool -L $i combined 6
>>>> ethtool -K $i ntuple on
>>>> ethtool -K $i gro on
>>>> ethtool -K $i tso on
>>>> done
>>>>
>>>> MEMLEAK: 0-2MB/s with some recycles
>>>> 0 MB/10sec
>>>> 0 MB/10sec
>>>> 0 MB/10sec
>>>> 0 MB/10sec
>>>> 0 MB/10sec
>>>> 0 MB/10sec
>>>> 1 MB/10sec
>>>> 0 MB/10sec
>>>> 2 MB/10sec
>>>> 0 MB/10sec
>>>> 2 MB/10sec
>>>> -1 MB/10sec
>>>> 0 MB/10sec
>>>> 2 MB/10sec
>>>> 0 MB/10sec
>>>> 2 MB/10sec
>>>> -1 MB/10sec
>>>> 1 MB/10sec
>>> This data doesn't tell me much of anything and isn't what I asked for.
>>> I don't see how the interrupt throttling rate would be associated with
>>> your memory leak other than possibly rate limiting it by rate limiting
>>> the traffic itself. Is there something that gave you the impression
>>> that interrupt rate was somehow involved?
>> more interrupts more leak
>>
>>>
>>> When we last talked I had asked if you could do a git bisect to find
>>> the memory leak and you said you would look into it. The most useful
>>> way to solve this would be to do a git bisect between your current
>>> kernel and the 4.11 kernel to find the point at which this started. If
>>> we can do that then fixing this becomes much simpler as we just have
>>> to fix the patch that introduced the issue.
>>>
>>> Also, I don't know it is you are using to determine that there is a
>>> memory leak. What tool is it you are using to do the tracking? Is
>>> there any specific form of traffic that is causing the leak? If you
>>> can't perform the bisection, any information you could provide that
>>> would allow me to do it would also be useful.
>> simple script
>>
>> mem1=`free -m | grep Mem: | awk '{print $3}'`
>> sleep 10
>> mem2=`free -m | grep Mem: | awk '{print $3}'`
>>
>> num=$((mem2 - mem1))
>> echo $num " MB/10sec"
>>
>>
>> There is nothing more that gets mem
>> there is only routed traffic from interface A to B
>> nothings takes mem
>> And memleaks only anchge when i change the rx/tx usecs for card
>>
>> What You need more ?
>>
>> imagine this is not my only prblem but many - i just want to help i
>> changed cards to i40e based only cause somebody rises a bug - and i
>> want to use i40e in feature - dont need them now - but maybee it is
>> good to help ppl to solwe some problems now if i can - before i will
>> use this cards ?
>> I try to use i40e before but there was bug covered by bug - and
>> nobody from e1000.sf can help me they just reply after year and
>> closing tickets with info about no activity but they have info in
>> reported bugs ... soooo what is this ? support center ? for me no .
>> If i want to help -= after a year response will be something like -
>> "dont care now" - cause i'v used other hw or sme hacks to repair
>> problem that should be sloved by intel
>>
>>>
>>> Thanks.
>>>
>>> - Alex
>>>
>>
>>
> What i can say more
>
>
> is that
> if:
> adaptive-rx off
> adaptive-tx off
> rx-usecs 10
> tx-usecs 10
>
> There is almost no memleak
>
> but i dont know if this is problem rx-usecx=tx-usecs - then no memleak
> or just lower numbers for rx/tx-usecs - are doing this
>
>
> But if You see my graphs You will see that less rx-usecs = less memleak
>
>
>
>
>
>
>
Also have parallel running same host with same HW with same mem only
difference is one have 8x 10G with ixgbe driver and second 8x with 8x i40e
one leaking one not
So really what You want more ?
^ permalink raw reply
* [PATCH net-next] ipv4: ipv4_default_advmss() should use route mtu
From: Eric Dumazet @ 2017-10-19 0:02 UTC (permalink / raw)
To: David Miller; +Cc: netdev
From: Eric Dumazet <edumazet@google.com>
ipv4_default_advmss() incorrectly uses the device MTU instead
of the route provided one. IPv6 has the proper behavior,
lets harmonize the two protocols.
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
net/ipv4/route.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index 4306db8273742f5e845635419c296823301c4a8b..bc40bd4111969f640603dd7c5b04fdb7fdcd4afd 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -1250,7 +1250,7 @@ static void set_class_tag(struct rtable *rt, u32 tag)
static unsigned int ipv4_default_advmss(const struct dst_entry *dst)
{
unsigned int header_size = sizeof(struct tcphdr) + sizeof(struct iphdr);
- unsigned int advmss = max_t(unsigned int, dst->dev->mtu - header_size,
+ unsigned int advmss = max_t(unsigned int, ipv4_mtu(dst) - header_size,
ip_rt_min_advmss);
return min(advmss, IPV4_MAX_PMTU - header_size);
^ permalink raw reply related
* Re: RFC(v2): Audit Kernel Container IDs
From: Richard Guy Briggs @ 2017-10-19 0:05 UTC (permalink / raw)
To: Casey Schaufler
Cc: cgroups-u79uwXL29TY76Z2rM5mHXA, Linux Containers, Linux API,
Linux Audit, Linux FS Devel, Linux Kernel,
Linux Network Development, mszeredi-H+wXaHxf7aLQT0dZR+AlfA,
Andy Lutomirski, jlayton-H+wXaHxf7aLQT0dZR+AlfA,
Carlos O'Donell, Al Viro, David Howells, Simo Sorce,
trondmy-7I+n7zu2hftEKMMhf/gKZA, Eric Paris, Serge E. Hallyn,
Eric W. Biederman
In-Reply-To: <81c15928-c445-fb8e-251c-bee566fbbf58-iSGtlc1asvQWG2LlvL+J4A@public.gmane.org>
On 2017-10-17 01:10, Casey Schaufler wrote:
> On 10/16/2017 5:33 PM, Richard Guy Briggs wrote:
> > On 2017-10-12 16:33, Casey Schaufler wrote:
> >> On 10/12/2017 7:14 AM, Richard Guy Briggs wrote:
> >>> Containers are a userspace concept. The kernel knows nothing of them.
> >>>
> >>> The Linux audit system needs a way to be able to track the container
> >>> provenance of events and actions. Audit needs the kernel's help to do
> >>> this.
> >>>
> >>> Since the concept of a container is entirely a userspace concept, a
> >>> registration from the userspace container orchestration system initiates
> >>> this. This will define a point in time and a set of resources
> >>> associated with a particular container with an audit container ID.
> >>>
> >>> The registration is a pseudo filesystem (proc, since PID tree already
> >>> exists) write of a u8[16] UUID representing the container ID to a file
> >>> representing a process that will become the first process in a new
> >>> container. This write might place restrictions on mount namespaces
> >>> required to define a container, or at least careful checking of
> >>> namespaces in the kernel to verify permissions of the orchestrator so it
> >>> can't change its own container ID. A bind mount of nsfs may be
> >>> necessary in the container orchestrator's mntNS.
> >>> Note: Use a 128-bit scalar rather than a string to make compares faster
> >>> and simpler.
> >>>
> >>> Require a new CAP_CONTAINER_ADMIN to be able to carry out the
> >>> registration.
> >> Hang on. If containers are a user space concept, how can
> >> you want CAP_CONTAINER_ANYTHING? If there's not such thing as
> >> a container, how can you be asking for a capability to manage
> >> them?
> > There is such a thing, but the kernel doesn't know about it yet.
>
> Then how can it be the kernel's place to control access to a
> container resource, that is, the containerID.
Ok, let me try to address your objections.
The kernel can know enough that if it is already set to not allow it to
be set again. Or if the user doesn't have permission to set it that the
user be denied this action. How is this different from loginuid and
sessionid?
>
> > This
> > same situation exists for loginuid and sessionid which are userspace
> > concepts that the kernel tracks for the convenience of userspace.
>
> Ah, no. Loginuid identifies a user, which is a kernel concept in
> that a user is defined by the uid.
This simple explanation doesn't help me. What makes that a kernel
concept? The fact that it is stored and compared in more than one
place?
> The session ID has well defined kernel semantics. You're trying to say
> that the containerID is an opaque value that is meaningless to the
> kernel, but you still want the kernel to protect it. How can the
> kernel know if it is protecting it correctly?
How so? A userspace process triggers this. Does the kernel know what
these values mean? Does it do anything with them other than report
them or allow audit to filter them? It is given some instructions on
how to treat it.
This is what we're trying to do with the containerID.
> > As
> > for its name, I'm not particularly picky, so if you don't like
> > CAP_CONTAINER_* then I'm fine with CAP_AUDIT_CONTAINERID. It really
> > needs to be distinct from CAP_AUDIT_WRITE and CAP_AUDIT_CONTROL since we
> > don't want to give the ability to set a containerID to any process that
> > is able to do audit logging (such as vsftpd) and similarly we don't want
> > to give the orchestrator the ability to control the setup of the audit
> > daemon.
>
> Sorry, but what aspect of the kernel security policy is this
> capability supposed to protect? That's what capabilities are
> for, not the undefined support of undefined user-space behavior.
Similarly, loginuids and sessionIDs are only used for audit tracking and
filtering.
> If it's audit behavior, you want CAP_AUDIT_CONTROL. If it's
> more than audit behavior you have to define what system security
> policy you're dealing with in order to pick the right capability.
It isn't audit behaviour (yet), it is audit reporting information, a
level above simply writing logs and a level below controlling daemon
behaviour.
> We get this request pretty regularly. "I need my own capability
> because I have a niche thing that isn't part of the system security
> policy but that is important!" Fit the containerID into the
> system security policy, and if that results in using CAP_SYS_ADMIN,
> oh well.
There's far too much piled in to CAP_SYS_ADMIN already, which is making
capabilites less and less useful. I realize that capabilities are
limited compared with netlink message types, but this falls in between
the abilities needed by CAP_AUDIT_CONTROL and CAP_AUDIT_WRITE.
I'll continue on Steve Grubb's comment...
> >>> At that time, record the target container's user-supplied
> >>> container identifier along with the target container's first process
> >>> (which may become the target container's "init" process) process ID
> >>> (referenced from the initial PID namespace), all namespace IDs (in the
> >>> form of a nsfs device number and inode number tuple) in a new auxilliary
> >>> record AUDIT_CONTAINER with a qualifying op=$action field.
> >>>
> >>> Issue a new auxilliary record AUDIT_CONTAINER_INFO for each valid
> >>> container ID present on an auditable action or event.
> >>>
> >>> Forked and cloned processes inherit their parent's container ID,
> >>> referenced in the process' task_struct.
> >>>
> >>> Mimic setns(2) and return an error if the process has already initiated
> >>> threading or forked since this registration should happen before the
> >>> process execution is started by the orchestrator and hence should not
> >>> yet have any threads or children. If this is deemed overly restrictive,
> >>> switch all threads and children to the new containerID.
> >>>
> >>> Trust the orchestrator to judiciously use and restrict CAP_CONTAINER_ADMIN.
> >>>
> >>> Log the creation of every namespace, inheriting/adding its spawning
> >>> process' containerID(s), if applicable. Include the spawning and
> >>> spawned namespace IDs (device and inode number tuples).
> >>> [AUDIT_NS_CREATE, AUDIT_NS_DESTROY] [clone(2), unshare(2), setns(2)]
> >>> Note: At this point it appears only network namespaces may need to track
> >>> container IDs apart from processes since incoming packets may cause an
> >>> auditable event before being associated with a process.
> >>>
> >>> Log the destruction of every namespace when it is no longer used by any
> >>> process, include the namespace IDs (device and inode number tuples).
> >>> [AUDIT_NS_DESTROY] [process exit, unshare(2), setns(2)]
> >>>
> >>> Issue a new auxilliary record AUDIT_NS_CHANGE listing (opt: op=$action)
> >>> the parent and child namespace IDs for any changes to a process'
> >>> namespaces. [setns(2)]
> >>> Note: It may be possible to combine AUDIT_NS_* record formats and
> >>> distinguish them with an op=$action field depending on the fields
> >>> required for each message type.
> >>>
> >>> When a container ceases to exist because the last process in that
> >>> container has exited and hence the last namespace has been destroyed and
> >>> its refcount dropping to zero, log the fact.
> >>> (This latter is likely needed for certification accountability.) A
> >>> container object may need a list of processes and/or namespaces.
> >>>
> >>> A namespace cannot directly migrate from one container to another but
> >>> could be assigned to a newly spawned container. A namespace can be
> >>> moved from one container to another indirectly by having that namespace
> >>> used in a second process in another container and then ending all the
> >>> processes in the first container.
> >>>
> >>> (v2)
> >>> - switch from u64 to u128 UUID
> >>> - switch from "signal" and "trigger" to "register"
> >>> - restrict registration to single process or force all threads and children into same container
> >>>
> >>> - RGB
> > - RGB
> >
> > --
> > Richard Guy Briggs <rgb-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > Sr. S/W Engineer, Kernel Security, Base Operating Systems
> > Remote, Ottawa, Red Hat Canada
> > IRC: rgb, SunRaycer
> > Voice: +1.647.777.2635, Internal: (81) 32635
> >
>
- RGB
--
Richard Guy Briggs <rgb-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Sr. S/W Engineer, Kernel Security, Base Operating Systems
Remote, Ottawa, Red Hat Canada
IRC: rgb, SunRaycer
Voice: +1.647.777.2635, Internal: (81) 32635
^ permalink raw reply
* Re: [PATCH net-next v2 2/2] net: ethernet: socionext: add AVE ethernet driver
From: Masahiro Yamada @ 2017-10-19 0:29 UTC (permalink / raw)
To: Kunihiko Hayashi
Cc: netdev-u79uwXL29TY76Z2rM5mHXA, Andrew Lunn, Florian Fainelli,
Rob Herring, Mark Rutland, linux-arm-kernel,
Linux Kernel Mailing List, devicetree-u79uwXL29TY76Z2rM5mHXA,
Masami Hiramatsu, Jassi Brar
In-Reply-To: <20171018192359.243C.4A936039-uWyLwvC0a2jby3iVrkZq2A@public.gmane.org>
2017-10-18 19:23 GMT+09:00 Kunihiko Hayashi <hayashi.kunihiko-uWyLwvC0a2jby3iVrkZq2A@public.gmane.org>:
> On Mon, 16 Oct 2017 00:08:21 +0900 <yamada.masahiro-uWyLwvC0a2jby3iVrkZq2A@public.gmane.org> wrote:
>> priv->rst = devm_reset_control_get_optional_shared(dev, NULL);
>> if (IS_ERR(priv->rst))
>> return PTR_ERR(priv->rst);
>
> The clk and reset are optional in the driver.
> Referring to your suggested method, I'll fix the part of clk and reset.
>
Why is clk optional?
--
Best Regards
Masahiro Yamada
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: RFC(v2): Audit Kernel Container IDs
From: Eric W. Biederman @ 2017-10-19 0:43 UTC (permalink / raw)
To: Aleksa Sarai
Cc: Paul Moore, James Bottomley, cgroups-u79uwXL29TY76Z2rM5mHXA,
mszeredi-H+wXaHxf7aLQT0dZR+AlfA, Andy Lutomirski,
jlayton-H+wXaHxf7aLQT0dZR+AlfA, Carlos O'Donell, API,
Linux Containers, Linux Kernel, Viro, David Howells,
Linux FS Devel, linux-audit-H+wXaHxf7aLQT0dZR+AlfA, Simo Sorce,
Development, Casey Schaufler, Eric Paris, Steve Grubb,
trondmy-7I+n7zu2hftEKMMhf/gKZA
In-Reply-To: <49752b6f-8a77-d1e5-8acb-5a1eed0a992c-l3A5Bk7waGM@public.gmane.org>
Aleksa Sarai <asarai-l3A5Bk7waGM@public.gmane.org> writes:
>>> The security implications are that anything that can change the label
>>> could also hide itself and its doings from the audit system and thus
>>> would be used as a means to evade detection. I actually think this
>>> means the label should be write once (once you've set it, you can't
>>> change it) ...
>>
>> Richard and I have talked about a write once approach, but the
>> thinking was that you may want to allow a nested container
>> orchestrator (Why? I don't know, but people always want to do the
>> craziest things.) and a write-once policy makes that impossible. If
>> we punt on the nested orchestrator, I believe we can seriously think
>> about a write-once policy to simplify things.
>
> Nested containers are a very widely used use-case (see LXC system containers,
> inside of which people run other container runtimes). So I would definitely
> consider it something that "needs to be supported in some way". While the LXC
> guys might be a *tad* crazy, the use-case isn't. :P
Of course some of that gets to running auditd inside a container which
we don't have yet either.
So I think to start it is perfectly fine to figure out the non-nested
case first and what makes sense there. Then to sort out the nested
container case.
The solution might be that a process gets at most one id per ``audit
namespace''.
Eric
^ permalink raw reply
* Re: [PATCH net-next v2 2/2] ipv6: remove from fib tree aged out RTF_CACHE dst
From: Martin KaFai Lau @ 2017-10-19 1:11 UTC (permalink / raw)
To: Paolo Abeni
Cc: netdev, David S. Miller, Wei Wang, Eric Dumazet,
Hannes Frederic Sowa
In-Reply-To: <12138554b307c66d0de7afd582ba83e027c383a2.1508358132.git.pabeni@redhat.com>
On Wed, Oct 18, 2017 at 08:23:32PM +0000, Paolo Abeni wrote:
> The commit 2b760fcf5cfb ("ipv6: hook up exception table to store
> dst cache") partially reverted 1e2ea8ad37be ("ipv6: set
> dst.obsolete when a cached route has expired").
>
> As a result, RTF_CACHE dst referenced outside the fib tree will
> not be removed until the next sernum change; dst_check() does not
> fail on aged-out dst, and dst->__refcnt can't decrease: the aged
> out dst will stay valid for a potentially unlimited time after the
> timeout expiration.
>
> This change explicitly removes RTF_CACHE dst from the fib tree when
> aged out. The rt6_remove_exception() logic will then obsolete the
> dst and other entities will drop the related reference on next
> dst_check().
>
> v1 -> v2:
> - do not touch dst.obsolete in rt6_remove_exception(), not needed
>
> Fixes: 2b760fcf5cfb ("ipv6: hook up exception table to store dst cache")
> Signed-off-by: Paolo Abeni <pabeni@redhat.com>
> ---
> net/ipv6/route.c | 7 +++++--
> 1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/net/ipv6/route.c b/net/ipv6/route.c
> index 5c27313803d2..a4a7f199120e 100644
> --- a/net/ipv6/route.c
> +++ b/net/ipv6/route.c
> @@ -1575,8 +1575,11 @@ static void rt6_age_examine_exception(struct rt6_exception_bucket *bucket,
> {
> struct rt6_info *rt = rt6_ex->rt6i;
>
> - if (atomic_read(&rt->dst.__refcnt) == 1 &&
Wei and I have discussed a few cases:
1) For PMTU rt, the dst.expires should have been set (to ip6_rt_mtu_expires)
and the next dst_check() should have returned NULL. The
ip6_pol_route() should not have returned this expired pmtu rt also.
The ip6_rt_mtu_expires should be used to decide if the rt is no
longer useful instead of gc_args->timeout.
We should not ignore __refcnt in this case. Otherwise, we are
punishing sk that is caching the rt since lastuse
is only updated during the route lookup.
2) The cases that we can ignore __refcnt are:
a) RTF_CACHE rt that has no expire set. I believe only redirect rt falls
into this bucket. In this case, we can use gc_args->timeout.
b) If there is an urgency to cleanup (like reaching ip6_rt_max_size)
or user force a flush.
> - time_after_eq(now, rt->dst.lastuse + gc_args->timeout)) {
> + /* we are pruning and obsoleting the exception route even if others
> + * have still reference to it, so that on next dst_check() such
> + * reference can be dropped
> + */
> + if (time_after_eq(now, rt->dst.lastuse + gc_args->timeout)) {
> RT6_TRACE("aging clone %p\n", rt);
> rt6_remove_exception(bucket, rt6_ex);
> return;
> --
> 2.13.6
>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox