* Re: [PATCH v2 2/4] dt-bindings: net: qcom: Add binding for shared mdio bus
From: Wang, Dongsheng @ 2018-09-18 8:47 UTC (permalink / raw)
To: Florian Fainelli, Andrew Lunn
Cc: timur@kernel.org, davem@davemloft.net, Zheng, Joey,
netdev@vger.kernel.org, devicetree@vger.kernel.org
In-Reply-To: <13018118-9883-4b26-9017-b324d745cc58@gmail.com>
On 2018/9/18 0:54, Florian Fainelli wrote:
> On 09/17/2018 09:47 AM, Wang, Dongsheng wrote:
>> On 9/17/2018 10:50 PM, Andrew Lunn wrote:
>>> On Mon, Sep 17, 2018 at 04:53:29PM +0800, Wang Dongsheng wrote:
>>>> This property copy from "ibm,emac.txt" to describe a shared MIDO bus.
>>>> Since emac include MDIO, so If the motherboard has more than one PHY
>>>> connected to an MDIO bus, this property will point to the MAC device
>>>> that has the MDIO bus.
>>>>
>>>> Signed-off-by: Wang Dongsheng <dongsheng.wang@hxt-semitech.com>
>>>> ---
>>>> V2: s/Since QDF2400 emac/Since emac/
>>>> ---
>>>> Documentation/devicetree/bindings/net/qcom-emac.txt | 4 ++++
>>>> 1 file changed, 4 insertions(+)
>>>>
>>>> diff --git a/Documentation/devicetree/bindings/net/qcom-emac.txt b/Documentation/devicetree/bindings/net/qcom-emac.txt
>>>> index 346e6c7f47b7..50db71771358 100644
>>>> --- a/Documentation/devicetree/bindings/net/qcom-emac.txt
>>>> +++ b/Documentation/devicetree/bindings/net/qcom-emac.txt
>>>> @@ -24,6 +24,9 @@ Internal PHY node:
>>>> The external phy child node:
>>>> - reg : The phy address
>>>>
>>>> +Optional properties:
>>>> +- mdio-device : Shared MIDO bus.
>>> Hi Dongsheng
>>>
>>> I don't see why you need this property. The ethernet interface has a
>>> phy-handle which points to a PHY. That is all you need to find the PHY.
>> phy-handle is description PHY address. This property is describing an
>> MDIO controller.
>> Each QCOM emac include an MDIO controller, normally each emac only
>> connect one
>> PHY device, but when all of the PHY devices mdio lines connect one MDIO
>> controller
>> that is included in EMAC, we need to share this MDIO controller for
>> others EMAC.
> If you want to describe the MDIO controller, then you embed a mdio
> subnode into your Ethernet MAC node:
>
> emac0: ethernet@feb20000 {
> mdio {
> #address-cells = <1>;
> #size-cells = <0>;
>
> phy0: ethernet-phy@0 {
> reg = <0>;
> };
> };
> };
>
> And then each Ethernet MAC controller refers to their appropriate PHY
> device tree node using a phy-handle property to point to either their
> own MDIO controller, or another MAC's MDIO controller.
Sorry, I do not understand how phy-handle point to MDIO controller,
because phy-handle is defined to point to a phy.
I suppose you mean:
mdio_node = of_get_parent(phy_node);
emac0: ethernet@feb20000 {
phy-handle = <&phy0>;
mdio {
#address-cells = <1>;
#size-cells = <0>;
phy0: ethernet-phy@0 {
reg = <0>;
};
phy1: ethernet-phy@1 {
reg = <1>;
};
};
};
emac1: ethernet@fexxxxxx {
phy-handle = <&phy1>;
};
emac2: ethernet@fexxxxxx {
phy-handle = <&phy2>;
mdio {
#address-cells = <1>;
#size-cells = <0>;
phy2: ethernet-phy@2 {
reg = <0>;
};
};
};
> The IBM Emac is a old (not to say bad) example and it does not use the
> PHY library and the standard Device Tree node property, please don't use
> it as a reference.
Ok, thanks.
Cheers,
Dongsheng
^ permalink raw reply
* pegged softirq and NAPI race (?)
From: Song Liu @ 2018-09-18 8:41 UTC (permalink / raw)
To: Networking
Cc: edumazet@google.com, jeffrey.t.kirsher@intel.com,
alexander.h.duyck@intel.com, Michael Chan, Kernel Team
We are debugging this issue that netconsole message triggers pegged softirq
(ksoftirqd taking 100% CPU for many seconds). We found this issue in
production with both bnxt and ixgbe, on a 4.11 based kernel. This is easily
reproducible with ixgbe on 4.11, and latest net/net-next (see [1] for more
detail).
After debugging for some time, we found that this issue is likely related
to 39e6c8208d7b ("net: solve a NAPI race"). After reverting this commit,
the steps described in [1] cannot reproduce the issue on ixgbe. Reverting
this commit also reduces the chances we hit the issue with bnxt (it still
happens with a lower rate).
I tried to fix this issue with relaxed variant (or older version) of
napi_schedule_prep() in netpoll, just like the one on napi_watchdog().
However, my tests do not always go as expected.
Please share your comments/suggestions on which direction shall we try
to fix this.
Thanks in advance!
Song
[1] https://www.spinics.net/lists/netdev/msg522328.html
^ permalink raw reply
* Re: [PATCH v2 net] net: aquantia: memory corruption on jumbo frames
From: Igor Russkikh @ 2018-09-18 8:28 UTC (permalink / raw)
To: David Miller; +Cc: nikita.danilov, netdev, f.gerold
In-Reply-To: <20180917.081520.1792531317539601050.davem@davemloft.net>
>> Fixes: 018423e90bee ("net: ethernet: aquantia: Add ring support code")
>>
>> Reported-by: Friedemann Gerold <f.gerold@b-c-s.de>
>> Reported-by: Michael Rauch <michael@rauch.be>
>> Signed-off-by: Friedemann Gerold <f.gerold@b-c-s.de>
>> Tested-by: Nikita Danilov <nikita.danilov@aquantia.com>
>> Signed-off-by: Igor Russkikh <igor.russkikh@aquantia.com>
>
> APplied and queued up for -stable.
Hi David, I see this was applied to net-next tree only.
Can we have it in net? We consider it as critical flaw.
BR, Igor
^ permalink raw reply
* Re: [PATCH] bonding: avoid repeated display of same link status change
From: Eric Dumazet @ 2018-09-18 14:00 UTC (permalink / raw)
To: Manish Kumar Singh, Eric Dumazet, netdev
Cc: Jay Vosburgh, Veaceslav Falico, Andy Gospodarek, David S. Miller,
linux-kernel
In-Reply-To: <78bf3185-fbd1-4a6c-ae32-70da0af2cbb4@default>
On 09/17/2018 10:05 PM, Manish Kumar Singh wrote:
>
>
>> -----Original Message-----
>> From: Eric Dumazet [mailto:eric.dumazet@gmail.com]
>> Sent: 17 सितम्बर 2018 20:08
>> To: Manish Kumar Singh; netdev@vger.kernel.org
>> Cc: Jay Vosburgh; Veaceslav Falico; Andy Gospodarek; David S. Miller; linux-
>> kernel@vger.kernel.org
>> Subject: Re: [PATCH] bonding: avoid repeated display of same link status
>> change
>>
>>
>>
>> On 09/17/2018 12:20 AM, mk.singh@oracle.com wrote:
>>> From: Manish Kumar Singh <mk.singh@oracle.com>
>>>
>>> When link status change needs to be committed and rtnl lock couldn't be
>>> taken, avoid redisplay of same link status change message.
>>>
>>> Signed-off-by: Manish Kumar Singh <mk.singh@oracle.com>
>>> ---
>>> drivers/net/bonding/bond_main.c | 6 ++++--
>>> include/net/bonding.h | 1 +
>>> 2 files changed, 5 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/net/bonding/bond_main.c
>> b/drivers/net/bonding/bond_main.c
>>> index 217b790d22ed..fb4e3aff1677 100644
>>> --- a/drivers/net/bonding/bond_main.c
>>> +++ b/drivers/net/bonding/bond_main.c
>>> @@ -2087,7 +2087,7 @@ static int bond_miimon_inspect(struct bonding
>> *bond)
>>> bond_propose_link_state(slave, BOND_LINK_FAIL);
>>> commit++;
>>> slave->delay = bond->params.downdelay;
>>> - if (slave->delay) {
>>> + if (slave->delay && !bond->rtnl_needed) {
>>> netdev_info(bond->dev, "link status down for
>> %sinterface %s, disabling it in %d ms\n",
>>> (BOND_MODE(bond) ==
>>> BOND_MODE_ACTIVEBACKUP) ?
>>> @@ -2127,7 +2127,7 @@ static int bond_miimon_inspect(struct bonding
>> *bond)
>>> commit++;
>>> slave->delay = bond->params.updelay;
>>>
>>> - if (slave->delay) {
>>> + if (slave->delay && !bond->rtnl_needed) {
>>> netdev_info(bond->dev, "link status up for
>> interface %s, enabling it in %d ms\n",
>>> slave->dev->name,
>>> ignore_updelay ? 0 :
>>> @@ -2301,9 +2301,11 @@ static void bond_mii_monitor(struct
>> work_struct *work)
>>> if (!rtnl_trylock()) {
>>> delay = 1;
>>> should_notify_peers = false;
>>> + bond->rtnl_needed = true;
>>
>> How can you set a shared variable with no synchronization ?
> Thanks Eric for reviewing the patch. rtnl_needed is not a shared variable, it is part of bonding structure, that is one per bonding driver instance. There can't be two parallel instances of bond_miimon_inspect for a single bonding driver instance at any given point of time. and only bond_miimon_inspect updates it. That’s why I think there is no need of any synchronization here.
>
>
If rtnl_trylock() can not grab RTNL,
there is no way the current thread can set the variable without a race, if the word including rtnl_needed is shared by other fields in the structure.
Your patch adds a subtle possibility of future bugs, even if it runs fine today.
Do not pave the way for future bugs, make your code robust, please.
^ permalink raw reply
* [PATCH net 5/5] net/smc: fix sizeof to int comparison
From: Ursula Braun @ 2018-09-18 13:46 UTC (permalink / raw)
To: davem; +Cc: netdev, linux-s390, schwidefsky, heiko.carstens, raspl,
linux-kernel
In-Reply-To: <20180918134638.90271-1-ubraun@linux.ibm.com>
From: YueHaibing <yuehaibing@huawei.com>
Comparing an int to a size, which is unsigned, causes the int to become
unsigned, giving the wrong result. kernel_sendmsg can return a negative
error code.
Signed-off-by: YueHaibing <yuehaibing@huawei.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
---
net/smc/smc_clc.c | 14 ++++++--------
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git a/net/smc/smc_clc.c b/net/smc/smc_clc.c
index 83aba9ade060..52241d679cc9 100644
--- a/net/smc/smc_clc.c
+++ b/net/smc/smc_clc.c
@@ -446,14 +446,12 @@ int smc_clc_send_proposal(struct smc_sock *smc, int smc_type,
vec[i++].iov_len = sizeof(trl);
/* due to the few bytes needed for clc-handshake this cannot block */
len = kernel_sendmsg(smc->clcsock, &msg, vec, i, plen);
- if (len < sizeof(pclc)) {
- if (len >= 0) {
- reason_code = -ENETUNREACH;
- smc->sk.sk_err = -reason_code;
- } else {
- smc->sk.sk_err = smc->clcsock->sk->sk_err;
- reason_code = -smc->sk.sk_err;
- }
+ if (len < 0) {
+ smc->sk.sk_err = smc->clcsock->sk->sk_err;
+ reason_code = -smc->sk.sk_err;
+ } else if (len < (int)sizeof(pclc)) {
+ reason_code = -ENETUNREACH;
+ smc->sk.sk_err = -reason_code;
}
return reason_code;
--
2.16.4
^ permalink raw reply related
* [PATCH net 4/5] net/smc: no urgent data check for listen sockets
From: Ursula Braun @ 2018-09-18 13:46 UTC (permalink / raw)
To: davem; +Cc: netdev, linux-s390, schwidefsky, heiko.carstens, raspl,
linux-kernel
In-Reply-To: <20180918134638.90271-1-ubraun@linux.ibm.com>
From: Karsten Graul <kgraul@linux.ibm.com>
Don't check a listen socket for pending urgent data in smc_poll().
Signed-off-by: Karsten Graul <kgraul@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
---
net/smc/af_smc.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
index 5c6d30eb4a71..015231789ed2 100644
--- a/net/smc/af_smc.c
+++ b/net/smc/af_smc.c
@@ -1566,9 +1566,9 @@ static __poll_t smc_poll(struct file *file, struct socket *sock,
mask |= EPOLLIN | EPOLLRDNORM | EPOLLRDHUP;
if (sk->sk_state == SMC_APPCLOSEWAIT1)
mask |= EPOLLIN;
+ if (smc->conn.urg_state == SMC_URG_VALID)
+ mask |= EPOLLPRI;
}
- if (smc->conn.urg_state == SMC_URG_VALID)
- mask |= EPOLLPRI;
}
return mask;
--
2.16.4
^ permalink raw reply related
* Re: [RFC/fix] Re: libbpf build broken on musl libc (Alpine Linux)
From: Arnaldo Carvalho de Melo @ 2018-09-18 13:15 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Arnaldo Carvalho de Melo, Jakub Kicinski, Daniel Borkmann,
Thomas Richter, Hendrik Brueckner, Jiri Olsa, Namhyung Kim,
linux-kernel, netdev
In-Reply-To: <20180918005256.7uutwda4s3ofoxpd@ast-mbp>
Em Mon, Sep 17, 2018 at 05:52:59PM -0700, Alexei Starovoitov escreveu:
> > +++ b/tools/lib/bpf/str_error.h
> > @@ -0,0 +1,6 @@
> > +// SPDX-License-Identifier: GPL-2.1
>
> LGPL-2.1 in the above?
Sure, will fix.
> The rest looks good to me.
Will that that as an Acked-by: you, ok?
> Should we take it via bpf-next tree?
No need, I have it already in perf/urgent, will push shortly to Ingo.
Thanks,
- Arnaldo
> If you feel there is an urgency to fix musl build, we can take it via bpf tree too.
>
> Jakub, thoughts? you've been messing with strerror last..
^ permalink raw reply
* Re: [PATCH net-next v2 1/2] net: stmmac: Rework coalesce timer and fix multi-queue races
From: Tal Gilboa @ 2018-09-18 7:02 UTC (permalink / raw)
To: Florian Fainelli, Jose Abreu, netdev
Cc: Jerome Brunet, Martin Blumenstingl, David S. Miller, Joao Pinto,
Giuseppe Cavallaro, Alexandre Torgue
In-Reply-To: <f0288def-ecf3-851b-fe81-12f0f79a061d@gmail.com>
On 9/10/2018 10:22 PM, Florian Fainelli wrote:
> On 09/10/2018 02:14 AM, Jose Abreu wrote:
>> This follows David Miller advice and tries to fix coalesce timer in
>> multi-queue scenarios.
>>
>> We are now using per-queue coalesce values and per-queue TX timer.
>>
>> Coalesce timer default values was changed to 1ms and the coalesce frames
>> to 25.
>>
>> Tested in B2B setup between XGMAC2 and GMAC5.
>
> Why not revert the entire features for this merge window and work on
> getting it to work over the next weeks/merge windows?
>
> The idea of using a timer to coalesce TX path when there is not a HW
> timer is a good idea and if this is made robust enough, you could even
> promote that as being a network stack library/feature that could be used
> by other drivers. In fact, this could be a great addition to the net DIM
> library (Tal, what do you think?)
Not sure it would be a natural fit. DIM doesn't know/care which type of
timer is used. Maybe the two mechanisms can work together with DIM
optimizes the number of frames to use for optimal coalescing. As the
timer is set to 1ms, DIM would have issues suggesting meaningful values
as it is designed for typical values of 10s on us.
>
> Here's a quick drive by review of things that appear wrong in the
> current driver (without your patches):
>
> - in stmmac_xmit(), in case we hit the !is_jumbo branch and we fail the
> DMA mapping, there is no timer cancellation, don't we want to abort the
> whole transmission?
>
> - stmmac_tx_clean() should probably use netif_lock_bh() to guard against
> the timer (soft IRQ context) and the the NAPI context (also soft IRQ)
> running in parallel on two different CPUs. This may not explain all
> problems, but these two things are fundamentally exclusive, because the
> timer is meant to emulate the interrupt after N packets, while NAPI
> executes when such a thing did actually occur
>
> - stmmac_poll() should cancel pending timer(s) if it was able to reclaim
> packets, likewise stmmac_tx_timer() should re-enable TX interrupts if it
> reclaimed packets, since TX interrupts could have been left disabled
> from a prior NAPI run. These could be considered optimizations, since
> you could leave the TX timer running all the time, just adjust the
> deadline (based on line rate, MTU, IPG, number of fragments and their
> respective length), worst case, both NAPI and the timer clean up your TX
> ring, so you should always have room to push more packets
>
>>
>> Signed-off-by: Jose Abreu <joabreu@synopsys.com>
>> Cc: Jerome Brunet <jbrunet@baylibre.com>
>> Cc: Martin Blumenstingl <martin.blumenstingl@googlemail.com>
>> Cc: David S. Miller <davem@davemloft.net>
>> Cc: Joao Pinto <jpinto@synopsys.com>
>> Cc: Giuseppe Cavallaro <peppe.cavallaro@st.com>
>> Cc: Alexandre Torgue <alexandre.torgue@st.com>
>> ---
>> Jerome,
>>
>> Can you please test if this one is okay ?
>>
>> Thanks and Best Regards,
>> Jose Miguel Abreu
>> ---
>> drivers/net/ethernet/stmicro/stmmac/common.h | 4 +-
>> drivers/net/ethernet/stmicro/stmmac/stmmac.h | 6 +-
>> drivers/net/ethernet/stmicro/stmmac/stmmac_main.c | 207 ++++++++++++++--------
>> 3 files changed, 135 insertions(+), 82 deletions(-)
>>
>> diff --git a/drivers/net/ethernet/stmicro/stmmac/common.h b/drivers/net/ethernet/stmicro/stmmac/common.h
>> index 1854f270ad66..b1b305f8f414 100644
>> --- a/drivers/net/ethernet/stmicro/stmmac/common.h
>> +++ b/drivers/net/ethernet/stmicro/stmmac/common.h
>> @@ -258,10 +258,10 @@ struct stmmac_safety_stats {
>> #define MAX_DMA_RIWT 0xff
>> #define MIN_DMA_RIWT 0x20
>> /* Tx coalesce parameters */
>> -#define STMMAC_COAL_TX_TIMER 40000
>> +#define STMMAC_COAL_TX_TIMER 1000
>> #define STMMAC_MAX_COAL_TX_TICK 100000
>> #define STMMAC_TX_MAX_FRAMES 256
>> -#define STMMAC_TX_FRAMES 64
>> +#define STMMAC_TX_FRAMES 25
>>
>> /* Packets types */
>> enum packets_types {
>> diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac.h b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
>> index c0a855b7ab3b..957030cfb833 100644
>> --- a/drivers/net/ethernet/stmicro/stmmac/stmmac.h
>> +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
>> @@ -48,6 +48,9 @@ struct stmmac_tx_info {
>>
>> /* Frequently used values are kept adjacent for cache effect */
>> struct stmmac_tx_queue {
>> + u32 tx_count_frames;
>> + int tx_timer_active;
>> + struct timer_list txtimer;
>> u32 queue_index;
>> struct stmmac_priv *priv_data;
>> struct dma_extended_desc *dma_etx ____cacheline_aligned_in_smp;
>> @@ -59,6 +62,7 @@ struct stmmac_tx_queue {
>> dma_addr_t dma_tx_phy;
>> u32 tx_tail_addr;
>> u32 mss;
>> + struct napi_struct napi ____cacheline_aligned_in_smp;
>> };
>>
>> struct stmmac_rx_queue {
>> @@ -109,14 +113,12 @@ struct stmmac_pps_cfg {
>>
>> struct stmmac_priv {
>> /* Frequently used values are kept adjacent for cache effect */
>> - u32 tx_count_frames;
>> u32 tx_coal_frames;
>> u32 tx_coal_timer;
>>
>> int tx_coalesce;
>> int hwts_tx_en;
>> bool tx_path_in_lpi_mode;
>> - struct timer_list txtimer;
>> bool tso;
>>
>> unsigned int dma_buf_sz;
>> diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
>> index 9f458bb16f2a..9809c2b319fe 100644
>> --- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
>> +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
>> @@ -148,6 +148,7 @@ static void stmmac_verify_args(void)
>> static void stmmac_disable_all_queues(struct stmmac_priv *priv)
>> {
>> u32 rx_queues_cnt = priv->plat->rx_queues_to_use;
>> + u32 tx_queues_cnt = priv->plat->tx_queues_to_use;
>> u32 queue;
>>
>> for (queue = 0; queue < rx_queues_cnt; queue++) {
>> @@ -155,6 +156,12 @@ static void stmmac_disable_all_queues(struct stmmac_priv *priv)
>>
>> napi_disable(&rx_q->napi);
>> }
>> +
>> + for (queue = 0; queue < tx_queues_cnt; queue++) {
>> + struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
>> +
>> + napi_disable(&tx_q->napi);
>> + }
>> }
>>
>> /**
>> @@ -164,6 +171,7 @@ static void stmmac_disable_all_queues(struct stmmac_priv *priv)
>> static void stmmac_enable_all_queues(struct stmmac_priv *priv)
>> {
>> u32 rx_queues_cnt = priv->plat->rx_queues_to_use;
>> + u32 tx_queues_cnt = priv->plat->tx_queues_to_use;
>> u32 queue;
>>
>> for (queue = 0; queue < rx_queues_cnt; queue++) {
>> @@ -171,6 +179,12 @@ static void stmmac_enable_all_queues(struct stmmac_priv *priv)
>>
>> napi_enable(&rx_q->napi);
>> }
>> +
>> + for (queue = 0; queue < tx_queues_cnt; queue++) {
>> + struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
>> +
>> + napi_enable(&tx_q->napi);
>> + }
>> }
>>
>> /**
>> @@ -1843,7 +1857,8 @@ static void stmmac_dma_operation_mode(struct stmmac_priv *priv)
>> * @queue: TX queue index
>> * Description: it reclaims the transmit resources after transmission completes.
>> */
>> -static void stmmac_tx_clean(struct stmmac_priv *priv, u32 queue)
>> +static int stmmac_tx_clean(struct stmmac_priv *priv, int limit, u32 queue,
>> + bool *more)
>> {
>> struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
>> unsigned int bytes_compl = 0, pkts_compl = 0;
>> @@ -1851,10 +1866,13 @@ static void stmmac_tx_clean(struct stmmac_priv *priv, u32 queue)
>>
>> netif_tx_lock(priv->dev);
>>
>> + if (more)
>> + *more = false;
>> +
>> priv->xstats.tx_clean++;
>>
>> entry = tx_q->dirty_tx;
>> - while (entry != tx_q->cur_tx) {
>> + while ((entry != tx_q->cur_tx) && (pkts_compl < limit)) {
>> struct sk_buff *skb = tx_q->tx_skbuff[entry];
>> struct dma_desc *p;
>> int status;
>> @@ -1937,7 +1955,13 @@ static void stmmac_tx_clean(struct stmmac_priv *priv, u32 queue)
>> stmmac_enable_eee_mode(priv);
>> mod_timer(&priv->eee_ctrl_timer, STMMAC_LPI_T(eee_timer));
>> }
>> +
>> + if (more && (tx_q->dirty_tx != tx_q->cur_tx))
>> + *more = true;
>> +
>> netif_tx_unlock(priv->dev);
>> +
>> + return pkts_compl;
>> }
>>
>> /**
>> @@ -2020,6 +2044,34 @@ static bool stmmac_safety_feat_interrupt(struct stmmac_priv *priv)
>> return false;
>> }
>>
>> +static int stmmac_napi_check(struct stmmac_priv *priv, u32 chan)
>> +{
>> + int status = stmmac_dma_interrupt_status(priv, priv->ioaddr,
>> + &priv->xstats, chan);
>> +
>> + if ((status & handle_rx) && (chan < priv->plat->rx_queues_to_use)) {
>> + struct stmmac_rx_queue *rx_q = &priv->rx_queue[chan];
>> +
>> + if (likely(napi_schedule_prep(&rx_q->napi))) {
>> + stmmac_disable_dma_irq(priv, priv->ioaddr, chan);
>> + __napi_schedule(&rx_q->napi);
>> + }
>> + } else {
>> + status &= ~handle_rx;
>> + }
>> +
>> + if ((status & handle_tx) && (chan < priv->plat->tx_queues_to_use)) {
>> + struct stmmac_tx_queue *tx_q = &priv->tx_queue[chan];
>> +
>> + if (likely(napi_schedule_prep(&tx_q->napi)))
>> + __napi_schedule(&tx_q->napi);
>> + } else {
>> + status &= ~handle_tx;
>> + }
>> +
>> + return status;
>> +}
>> +
>> /**
>> * stmmac_dma_interrupt - DMA ISR
>> * @priv: driver private structure
>> @@ -2034,57 +2086,14 @@ static void stmmac_dma_interrupt(struct stmmac_priv *priv)
>> u32 channels_to_check = tx_channel_count > rx_channel_count ?
>> tx_channel_count : rx_channel_count;
>> u32 chan;
>> - bool poll_scheduled = false;
>> int status[max_t(u32, MTL_MAX_TX_QUEUES, MTL_MAX_RX_QUEUES)];
>>
>> /* Make sure we never check beyond our status buffer. */
>> if (WARN_ON_ONCE(channels_to_check > ARRAY_SIZE(status)))
>> channels_to_check = ARRAY_SIZE(status);
>>
>> - /* Each DMA channel can be used for rx and tx simultaneously, yet
>> - * napi_struct is embedded in struct stmmac_rx_queue rather than in a
>> - * stmmac_channel struct.
>> - * Because of this, stmmac_poll currently checks (and possibly wakes)
>> - * all tx queues rather than just a single tx queue.
>> - */
>> for (chan = 0; chan < channels_to_check; chan++)
>> - status[chan] = stmmac_dma_interrupt_status(priv, priv->ioaddr,
>> - &priv->xstats, chan);
>> -
>> - for (chan = 0; chan < rx_channel_count; chan++) {
>> - if (likely(status[chan] & handle_rx)) {
>> - struct stmmac_rx_queue *rx_q = &priv->rx_queue[chan];
>> -
>> - if (likely(napi_schedule_prep(&rx_q->napi))) {
>> - stmmac_disable_dma_irq(priv, priv->ioaddr, chan);
>> - __napi_schedule(&rx_q->napi);
>> - poll_scheduled = true;
>> - }
>> - }
>> - }
>> -
>> - /* If we scheduled poll, we already know that tx queues will be checked.
>> - * If we didn't schedule poll, see if any DMA channel (used by tx) has a
>> - * completed transmission, if so, call stmmac_poll (once).
>> - */
>> - if (!poll_scheduled) {
>> - for (chan = 0; chan < tx_channel_count; chan++) {
>> - if (status[chan] & handle_tx) {
>> - /* It doesn't matter what rx queue we choose
>> - * here. We use 0 since it always exists.
>> - */
>> - struct stmmac_rx_queue *rx_q =
>> - &priv->rx_queue[0];
>> -
>> - if (likely(napi_schedule_prep(&rx_q->napi))) {
>> - stmmac_disable_dma_irq(priv,
>> - priv->ioaddr, chan);
>> - __napi_schedule(&rx_q->napi);
>> - }
>> - break;
>> - }
>> - }
>> - }
>> + status[chan] = stmmac_napi_check(priv, chan);
>>
>> for (chan = 0; chan < tx_channel_count; chan++) {
>> if (unlikely(status[chan] & tx_hard_error_bump_tc)) {
>> @@ -2241,13 +2250,11 @@ static int stmmac_init_dma_engine(struct stmmac_priv *priv)
>> */
>> static void stmmac_tx_timer(struct timer_list *t)
>> {
>> - struct stmmac_priv *priv = from_timer(priv, t, txtimer);
>> - u32 tx_queues_count = priv->plat->tx_queues_to_use;
>> - u32 queue;
>> + struct stmmac_tx_queue *tx_q = from_timer(tx_q, t, txtimer);
>>
>> - /* let's scan all the tx queues */
>> - for (queue = 0; queue < tx_queues_count; queue++)
>> - stmmac_tx_clean(priv, queue);
>> + if (likely(napi_schedule_prep(&tx_q->napi)))
>> + __napi_schedule(&tx_q->napi);
>> + tx_q->tx_timer_active = 0;
>> }
>>
>> /**
>> @@ -2260,11 +2267,17 @@ static void stmmac_tx_timer(struct timer_list *t)
>> */
>> static void stmmac_init_tx_coalesce(struct stmmac_priv *priv)
>> {
>> + u32 tx_channel_count = priv->plat->tx_queues_to_use;
>> + u32 chan;
>> +
>> priv->tx_coal_frames = STMMAC_TX_FRAMES;
>> priv->tx_coal_timer = STMMAC_COAL_TX_TIMER;
>> - timer_setup(&priv->txtimer, stmmac_tx_timer, 0);
>> - priv->txtimer.expires = STMMAC_COAL_TIMER(priv->tx_coal_timer);
>> - add_timer(&priv->txtimer);
>> +
>> + for (chan = 0; chan < tx_channel_count; chan++) {
>> + struct stmmac_tx_queue *tx_q = &priv->tx_queue[chan];
>> +
>> + timer_setup(&tx_q->txtimer, stmmac_tx_timer, 0);
>> + }
>> }
>>
>> static void stmmac_set_rings_length(struct stmmac_priv *priv)
>> @@ -2592,6 +2605,7 @@ static void stmmac_hw_teardown(struct net_device *dev)
>> static int stmmac_open(struct net_device *dev)
>> {
>> struct stmmac_priv *priv = netdev_priv(dev);
>> + u32 chan;
>> int ret;
>>
>> stmmac_check_ether_addr(priv);
>> @@ -2688,7 +2702,9 @@ static int stmmac_open(struct net_device *dev)
>> if (dev->phydev)
>> phy_stop(dev->phydev);
>>
>> - del_timer_sync(&priv->txtimer);
>> + for (chan = 0; chan < priv->plat->tx_queues_to_use; chan++)
>> + del_timer_sync(&priv->tx_queue[chan].txtimer);
>> +
>> stmmac_hw_teardown(dev);
>> init_error:
>> free_dma_desc_resources(priv);
>> @@ -2708,6 +2724,7 @@ static int stmmac_open(struct net_device *dev)
>> static int stmmac_release(struct net_device *dev)
>> {
>> struct stmmac_priv *priv = netdev_priv(dev);
>> + u32 chan;
>>
>> if (priv->eee_enabled)
>> del_timer_sync(&priv->eee_ctrl_timer);
>> @@ -2722,7 +2739,8 @@ static int stmmac_release(struct net_device *dev)
>>
>> stmmac_disable_all_queues(priv);
>>
>> - del_timer_sync(&priv->txtimer);
>> + for (chan = 0; chan < priv->plat->tx_queues_to_use; chan++)
>> + del_timer_sync(&priv->tx_queue[chan].txtimer);
>>
>> /* Free the IRQ lines */
>> free_irq(dev->irq, dev);
>> @@ -2936,14 +2954,11 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
>> priv->xstats.tx_tso_nfrags += nfrags;
>>
>> /* Manage tx mitigation */
>> - priv->tx_count_frames += nfrags + 1;
>> - if (likely(priv->tx_coal_frames > priv->tx_count_frames)) {
>> - mod_timer(&priv->txtimer,
>> - STMMAC_COAL_TIMER(priv->tx_coal_timer));
>> - } else {
>> - priv->tx_count_frames = 0;
>> + tx_q->tx_count_frames += nfrags + 1;
>> + if (priv->tx_coal_frames <= tx_q->tx_count_frames) {
>> stmmac_set_tx_ic(priv, desc);
>> priv->xstats.tx_set_ic_bit++;
>> + tx_q->tx_count_frames = 0;
>> }
>>
>> skb_tx_timestamp(skb);
>> @@ -2994,6 +3009,12 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
>>
>> stmmac_set_tx_tail_ptr(priv, priv->ioaddr, tx_q->tx_tail_addr, queue);
>>
>> + if (priv->tx_coal_timer && !tx_q->tx_timer_active) {
>> + tx_q->tx_timer_active = 1;
>> + mod_timer(&tx_q->txtimer,
>> + STMMAC_COAL_TIMER(priv->tx_coal_timer));
>> + }
>> +
>> return NETDEV_TX_OK;
>>
>> dma_map_err:
>> @@ -3146,14 +3167,11 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
>> * This approach takes care about the fragments: desc is the first
>> * element in case of no SG.
>> */
>> - priv->tx_count_frames += nfrags + 1;
>> - if (likely(priv->tx_coal_frames > priv->tx_count_frames)) {
>> - mod_timer(&priv->txtimer,
>> - STMMAC_COAL_TIMER(priv->tx_coal_timer));
>> - } else {
>> - priv->tx_count_frames = 0;
>> + tx_q->tx_count_frames += nfrags + 1;
>> + if (priv->tx_coal_frames <= tx_q->tx_count_frames) {
>> stmmac_set_tx_ic(priv, desc);
>> priv->xstats.tx_set_ic_bit++;
>> + tx_q->tx_count_frames = 0;
>> }
>>
>> skb_tx_timestamp(skb);
>> @@ -3199,8 +3217,15 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
>> netdev_tx_sent_queue(netdev_get_tx_queue(dev, queue), skb->len);
>>
>> stmmac_enable_dma_transmission(priv, priv->ioaddr);
>> +
>> stmmac_set_tx_tail_ptr(priv, priv->ioaddr, tx_q->tx_tail_addr, queue);
>>
>> + if (priv->tx_coal_timer && !tx_q->tx_timer_active) {
>> + tx_q->tx_timer_active = 1;
>> + mod_timer(&tx_q->txtimer,
>> + STMMAC_COAL_TIMER(priv->tx_coal_timer));
>> + }
>> +
>> return NETDEV_TX_OK;
>>
>> dma_map_err:
>> @@ -3514,27 +3539,41 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
>> * Description :
>> * To look at the incoming frames and clear the tx resources.
>> */
>> -static int stmmac_poll(struct napi_struct *napi, int budget)
>> +static int stmmac_rx_poll(struct napi_struct *napi, int budget)
>> {
>> struct stmmac_rx_queue *rx_q =
>> container_of(napi, struct stmmac_rx_queue, napi);
>> struct stmmac_priv *priv = rx_q->priv_data;
>> - u32 tx_count = priv->plat->tx_queues_to_use;
>> u32 chan = rx_q->queue_index;
>> int work_done = 0;
>> - u32 queue;
>>
>> priv->xstats.napi_poll++;
>>
>> - /* check all the queues */
>> - for (queue = 0; queue < tx_count; queue++)
>> - stmmac_tx_clean(priv, queue);
>> -
>> work_done = stmmac_rx(priv, budget, rx_q->queue_index);
>> + if (work_done < budget && napi_complete_done(napi, work_done))
>> + stmmac_enable_dma_irq(priv, priv->ioaddr, chan);
>> +
>> + return work_done;
>> +}
>> +
>> +static int stmmac_tx_poll(struct napi_struct *napi, int budget)
>> +{
>> + struct stmmac_tx_queue *tx_q =
>> + container_of(napi, struct stmmac_tx_queue, napi);
>> + struct stmmac_priv *priv = tx_q->priv_data;
>> + u32 chan = tx_q->queue_index;
>> + int work_done = 0;
>> + bool more;
>> +
>> + priv->xstats.napi_poll++;
>> +
>> + work_done = stmmac_tx_clean(priv, budget, chan, &more);
>> if (work_done < budget) {
>> napi_complete_done(napi, work_done);
>> - stmmac_enable_dma_irq(priv, priv->ioaddr, chan);
>> + if (more)
>> + napi_reschedule(napi);
>> }
>> +
>> return work_done;
>> }
>>
>> @@ -4325,10 +4364,17 @@ int stmmac_dvr_probe(struct device *device,
>> for (queue = 0; queue < priv->plat->rx_queues_to_use; queue++) {
>> struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
>>
>> - netif_napi_add(ndev, &rx_q->napi, stmmac_poll,
>> + netif_napi_add(ndev, &rx_q->napi, stmmac_rx_poll,
>> (8 * priv->plat->rx_queues_to_use));
>> }
>>
>> + for (queue = 0; queue < priv->plat->tx_queues_to_use; queue++) {
>> + struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
>> +
>> + netif_napi_add(ndev, &tx_q->napi, stmmac_tx_poll,
>> + (8 * priv->plat->tx_queues_to_use));
>> + }
>> +
>> mutex_init(&priv->lock);
>>
>> /* If a specific clk_csr value is passed from the platform
>> @@ -4377,6 +4423,11 @@ int stmmac_dvr_probe(struct device *device,
>>
>> netif_napi_del(&rx_q->napi);
>> }
>> + for (queue = 0; queue < priv->plat->tx_queues_to_use; queue++) {
>> + struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
>> +
>> + netif_napi_del(&tx_q->napi);
>> + }
>> error_hw_init:
>> destroy_workqueue(priv->wq);
>> error_wq:
>>
>
>
^ permalink raw reply
* Re: [PATCH 1/2] netlink: add NLA_REJECT policy type
From: Jamal Hadi Salim @ 2018-09-18 12:34 UTC (permalink / raw)
To: Johannes Berg, Marcelo Ricardo Leitner, Michal Kubecek
Cc: linux-wireless-u79uwXL29TY76Z2rM5mHXA,
netdev-u79uwXL29TY76Z2rM5mHXA, jbenc-H+wXaHxf7aLQT0dZR+AlfA
In-Reply-To: <1537177132.2957.6.camel-cdvu00un1VgdHxzADdlk8Q@public.gmane.org>
On 2018-09-17 5:38 a.m., Johannes Berg wrote:
> On Thu, 2018-09-13 at 18:58 -0300, Marcelo Ricardo Leitner wrote:
>
[..]
>
> So in one case I was thinking of, there are some fields that simply
> cannot be used for input, they're only used for output. > But it may not
> always be obvious to somebody using the API. Thus, I think it makes
> sense to instruct the kernel to reject that, so that whoever gets
> confused has immediate feedback that their usage is wrong. If we ignore
> that, they may not realize their error immediately.
>
> I think the ethtool case is similar: you can read and write some fields,
> and only read others - but if you try to write the read-only fields
> would you prefer to be told "sorry, this is not possible" vs. it being
> silently ignored? I'd definitely prefer the former.
>
>> Maybe it would be better to have NLA_IGNORE instead? </idea>
>
> I don't think so, it doesn't give any feedback to the application author
> that they're doing something wrong.
>
Maybe time to introduce kernel side access-control flags?
Read/Write permissions for example. Attrs marked as read only
(in the kernel) cannot be written to.
cheers,
jamal
^ permalink raw reply
* Re: [PATCH net-next] ravb: do not write 1 to reserved bits
From: Simon Horman @ 2018-09-18 6:56 UTC (permalink / raw)
To: Sergei Shtylyov
Cc: David Miller, Magnus Damm, netdev, linux-renesas-soc,
Kazuya Mizuguchi
In-Reply-To: <12758acf-6444-47eb-ad74-170034959bd7@cogentembedded.com>
On Mon, Sep 17, 2018 at 08:58:52PM +0300, Sergei Shtylyov wrote:
> On 09/17/2018 06:19 PM, Simon Horman wrote:
>
> > From: Kazuya Mizuguchi <kazuya.mizuguchi.ks@renesas.com>
> >
> > EtherAVB hardware requires 0 to be written to status register bits in
> > order to clear them, however, care must be taken not to:
> >
> > 1. Clear other bits, by writing zero to them
> > 2. Write one to reserved bits
> >
> > This patch corrects the ravb driver with respect to the second point above.
> > This is done by defining reserved bit masks for the affected registers and,
> > after auditing the code, ensure all sites that may write a one to a
> > reserved bit use are suitably masked.
> >
> > Signed-off-by: Kazuya Mizuguchi <kazuya.mizuguchi.ks@renesas.com>
> > Signed-off-by: Simon Horman <horms+renesas@verge.net.au>
> > ---
> > v2 [Simon Horman]
> > * Cover ravb_timestamp_interrupt() by this change
> > * Use enum value rather than #define for reserved masks
> > * Reword changelog
> >
> > v1 [Kazuya Mizuguchi]
> > ---
> > drivers/net/ethernet/renesas/ravb.h | 6 ++++++
> > drivers/net/ethernet/renesas/ravb_main.c | 11 ++++++-----
> > drivers/net/ethernet/renesas/ravb_ptp.c | 2 +-
> > 3 files changed, 13 insertions(+), 6 deletions(-)
> >
> > diff --git a/drivers/net/ethernet/renesas/ravb.h b/drivers/net/ethernet/renesas/ravb.h
> > index 1470fc12282b..bca219edcf94 100644
> > --- a/drivers/net/ethernet/renesas/ravb.h
> > +++ b/drivers/net/ethernet/renesas/ravb.h
> > @@ -428,6 +428,7 @@ enum EIS_BIT {
> > EIS_CULF1 = 0x00000080,
> > EIS_TFFF = 0x00000100,
> > EIS_QFS = 0x00010000,
> > + EIS_RESERVED = (u32)(GENMASK(31, 17) | GENMASK(15, 11)),
>
> Are you sure those (u32) casts are necessary? Happily builds in both 32- and 64-bit
> mode without them...
>
> [...]
> > @@ -528,6 +530,7 @@ enum RIS2_BIT {
> > RIS2_QFF16 = 0x00010000,
> > RIS2_QFF17 = 0x00020000,
> > RIS2_RFFF = 0x80000000,
> > + RIS2_RESERVED = (u32)GENMASK_ULL(30, 18),
>
> Why GENMASK_ULL() suddenly? Doesn't seem needed at all...
>
> [...]
> > @@ -544,6 +547,8 @@ enum TIS_BIT {
> > TIS_FTF1 = 0x00000002, /* Undocumented? */
> > TIS_TFUF = 0x00000100,
> > TIS_TFWF = 0x00000200,
> > + TIS_RESERVED = (u32)(GENMASK_ULL(31, 20) | GENMASK_ULL(15, 12) | \
> > + GENMASK_ULL(7, 4))
>
> Same question.
Thanks Sergei,
I agree GENMASK() without any cast should be sufficient in all
of the above cases. I'll respin accordingly.
^ permalink raw reply
* Re: linux-next: manual merge of the net-next tree with the net tree
From: Daniel Borkmann @ 2018-09-18 12:08 UTC (permalink / raw)
To: Stephen Rothwell, Vakul Garg
Cc: David Miller, Networking, Linux-Next Mailing List,
Linux Kernel Mailing List, davejwatson@fb.com, doronrk@fb.com
In-Reply-To: <20180918214814.4eae366d@canb.auug.org.au>
On 09/18/2018 01:48 PM, Stephen Rothwell wrote:
> Hi all,
>
> On Tue, 18 Sep 2018 10:17:03 +0000 Vakul Garg <vakul.garg@nxp.com> wrote:
>>
>> Got it.
>> Thanks for the guidance.
>
> So, what should I remove? ;-)
My (very own personal) preference in general would be that we test / assert
the kernel behavior that exists /today/, meaning once we implement support
for multi-record peek we add the corresponding test case along with that
code. Fwiw, this practice would be consistent with the rest of the kernel
selftests development model we have under net (& bpf).
Thanks,
Daniel
^ permalink raw reply
* Re: linux-next: manual merge of the net-next tree with the net tree
From: Stephen Rothwell @ 2018-09-18 11:48 UTC (permalink / raw)
To: Vakul Garg
Cc: Daniel Borkmann, David Miller, Networking,
Linux-Next Mailing List, Linux Kernel Mailing List,
davejwatson@fb.com, doronrk@fb.com
In-Reply-To: <DB7PR04MB425225AD548F3A3040250EF48B1D0@DB7PR04MB4252.eurprd04.prod.outlook.com>
[-- Attachment #1: Type: text/plain, Size: 196 bytes --]
Hi all,
On Tue, 18 Sep 2018 10:17:03 +0000 Vakul Garg <vakul.garg@nxp.com> wrote:
>
> Got it.
> Thanks for the guidance.
So, what should I remove? ;-)
--
Cheers,
Stephen Rothwell
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* [PATCH][net-next] netlink: avoid to allocate full skb when sending to many devices
From: Li RongQing @ 2018-09-18 5:26 UTC (permalink / raw)
To: netdev
if skb->head is vmalloc address, when this skb is delivered, full
allocation for this skb is required, if there are many devices,
the full allocation will be called for every devices
now using the first time allocated skb when iterate other devices
to send, reduce full allocation and speedup deliver.
Signed-off-by: Zhang Yu <zhangyu31@baidu.com>
Signed-off-by: Li RongQing <lirongqing@baidu.com>
---
net/netlink/af_netlink.c | 14 ++++++++------
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/net/netlink/af_netlink.c b/net/netlink/af_netlink.c
index e3a0538ec0be..095b99e3c1fb 100644
--- a/net/netlink/af_netlink.c
+++ b/net/netlink/af_netlink.c
@@ -278,11 +278,11 @@ static bool netlink_filter_tap(const struct sk_buff *skb)
return false;
}
-static int __netlink_deliver_tap_skb(struct sk_buff *skb,
+static int __netlink_deliver_tap_skb(struct sk_buff **skb,
struct net_device *dev)
{
struct sk_buff *nskb;
- struct sock *sk = skb->sk;
+ struct sock *sk = (*skb)->sk;
int ret = -ENOMEM;
if (!net_eq(dev_net(dev), sock_net(sk)))
@@ -290,10 +290,12 @@ static int __netlink_deliver_tap_skb(struct sk_buff *skb,
dev_hold(dev);
- if (is_vmalloc_addr(skb->head))
- nskb = netlink_to_full_skb(skb, GFP_ATOMIC);
+ if (is_vmalloc_addr((*skb)->head)) {
+ nskb = netlink_to_full_skb(*skb, GFP_ATOMIC);
+ *skb = nskb;
+ }
else
- nskb = skb_clone(skb, GFP_ATOMIC);
+ nskb = skb_clone(*skb, GFP_ATOMIC);
if (nskb) {
nskb->dev = dev;
nskb->protocol = htons((u16) sk->sk_protocol);
@@ -318,7 +320,7 @@ static void __netlink_deliver_tap(struct sk_buff *skb, struct netlink_tap_net *n
return;
list_for_each_entry_rcu(tmp, &nn->netlink_tap_all, list) {
- ret = __netlink_deliver_tap_skb(skb, tmp->dev);
+ ret = __netlink_deliver_tap_skb(&skb, tmp->dev);
if (unlikely(ret))
break;
}
--
2.16.2
^ permalink raw reply related
* Re: [bpf PATCH v3 3/3] bpf: test_maps, only support ESTABLISHED socks
From: Y Song @ 2018-09-18 5:22 UTC (permalink / raw)
To: John Fastabend; +Cc: Eric Dumazet, Alexei Starovoitov, Daniel Borkmann, netdev
In-Reply-To: <20180918043647.14911.2812.stgit@john-Precision-Tower-5810>
On Mon, Sep 17, 2018 at 9:38 PM John Fastabend <john.fastabend@gmail.com> wrote:
>
> Ensure that sockets added to a sock{map|hash} that is not in the
> ESTABLISHED state is rejected.
>
> Fixes: 1aa12bdf1bfb ("bpf: sockmap, add sock close() hook to remove socks")
> Signed-off-by: John Fastabend <john.fastabend@gmail.com>
Acked-by: Yonghong Song <yhs@fb.com>
^ permalink raw reply
* Re: [bpf PATCH v3 2/3] bpf: sockmap, fix transition through disconnect without close
From: Y Song @ 2018-09-18 5:21 UTC (permalink / raw)
To: John Fastabend; +Cc: Eric Dumazet, Alexei Starovoitov, Daniel Borkmann, netdev
In-Reply-To: <20180918043642.14911.62447.stgit@john-Precision-Tower-5810>
On Mon, Sep 17, 2018 at 9:39 PM John Fastabend <john.fastabend@gmail.com> wrote:
>
> It is possible (via shutdown()) for TCP socks to go trough TCP_CLOSE
> state via tcp_disconnect() without actually calling tcp_close which
> would then call our bpf_tcp_close() callback. Because of this a user
> could disconnect a socket then put it in a LISTEN state which would
> break our assumptions about sockets always being ESTABLISHED state.
>
> To resolve this rely on the unhash hook, which is called in the
> disconnect case, to remove the sock from the sockmap.
>
> Reported-by: Eric Dumazet <edumazet@google.com>
> Fixes: 1aa12bdf1bfb ("bpf: sockmap, add sock close() hook to remove socks")
> Signed-off-by: John Fastabend <john.fastabend@gmail.com>
> ---
> 0 files changed
>
> diff --git a/kernel/bpf/sockmap.c b/kernel/bpf/sockmap.c
> index 1f97b55..5680d65 100644
> --- a/kernel/bpf/sockmap.c
> +++ b/kernel/bpf/sockmap.c
> @@ -132,6 +132,7 @@ struct smap_psock {
> struct work_struct gc_work;
>
> struct proto *sk_proto;
> + void (*save_unhash)(struct sock *sk);
> void (*save_close)(struct sock *sk, long timeout);
> void (*save_data_ready)(struct sock *sk);
> void (*save_write_space)(struct sock *sk);
> @@ -143,6 +144,7 @@ static int bpf_tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
> static int bpf_tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);
> static int bpf_tcp_sendpage(struct sock *sk, struct page *page,
> int offset, size_t size, int flags);
> +static void bpf_tcp_unhash(struct sock *sk);
> static void bpf_tcp_close(struct sock *sk, long timeout);
>
> static inline struct smap_psock *smap_psock_sk(const struct sock *sk)
> @@ -184,6 +186,7 @@ static void build_protos(struct proto prot[SOCKMAP_NUM_CONFIGS],
> struct proto *base)
> {
> prot[SOCKMAP_BASE] = *base;
> + prot[SOCKMAP_BASE].unhash = bpf_tcp_unhash;
> prot[SOCKMAP_BASE].close = bpf_tcp_close;
> prot[SOCKMAP_BASE].recvmsg = bpf_tcp_recvmsg;
> prot[SOCKMAP_BASE].stream_memory_read = bpf_tcp_stream_read;
> @@ -217,6 +220,7 @@ static int bpf_tcp_init(struct sock *sk)
> return -EBUSY;
> }
>
> + psock->save_unhash = sk->sk_prot->unhash;
> psock->save_close = sk->sk_prot->close;
> psock->sk_proto = sk->sk_prot;
>
> @@ -305,30 +309,12 @@ static struct smap_psock_map_entry *psock_map_pop(struct sock *sk,
> return e;
> }
>
> -static void bpf_tcp_close(struct sock *sk, long timeout)
> +static void bpf_tcp_remove(struct sock *sk, struct smap_psock *psock)
> {
> - void (*close_fun)(struct sock *sk, long timeout);
> struct smap_psock_map_entry *e;
> struct sk_msg_buff *md, *mtmp;
> - struct smap_psock *psock;
> struct sock *osk;
>
> - lock_sock(sk);
> - rcu_read_lock();
> - psock = smap_psock_sk(sk);
> - if (unlikely(!psock)) {
> - rcu_read_unlock();
> - release_sock(sk);
> - return sk->sk_prot->close(sk, timeout);
> - }
> -
> - /* The psock may be destroyed anytime after exiting the RCU critial
> - * section so by the time we use close_fun the psock may no longer
> - * be valid. However, bpf_tcp_close is called with the sock lock
> - * held so the close hook and sk are still valid.
> - */
> - close_fun = psock->save_close;
> -
> if (psock->cork) {
> free_start_sg(psock->sock, psock->cork, true);
> kfree(psock->cork);
> @@ -379,6 +365,42 @@ static void bpf_tcp_close(struct sock *sk, long timeout)
> kfree(e);
> e = psock_map_pop(sk, psock);
> }
> +}
> +
> +static void bpf_tcp_unhash(struct sock *sk)
> +{
> + void (*unhash_fun)(struct sock *sk);
> + struct smap_psock *psock;
> +
> + rcu_read_lock();
> + psock = smap_psock_sk(sk);
> + if (unlikely(!psock)) {
> + rcu_read_unlock();
> + if (sk->sk_prot->unhash)
> + return sk->sk_prot->unhash(sk);
> + return;
Nit: the above code can be
if (sk->sk_prot->unhash)
sk->sk_prot->unhash(sk);
return;
Other than this,
Acked-by: Yonghong Song <yhs@fb.com>
> + }
> + unhash_fun = psock->save_unhash;
> + bpf_tcp_remove(sk, psock);
> + rcu_read_unlock();
> + unhash_fun(sk);
> +}
> +
> +static void bpf_tcp_close(struct sock *sk, long timeout)
> +{
> + void (*close_fun)(struct sock *sk, long timeout);
> + struct smap_psock *psock;
> +
> + lock_sock(sk);
> + rcu_read_lock();
> + psock = smap_psock_sk(sk);
> + if (unlikely(!psock)) {
> + rcu_read_unlock();
> + release_sock(sk);
> + return sk->sk_prot->close(sk, timeout);
> + }
> + close_fun = psock->save_close;
> + bpf_tcp_remove(sk, psock);
> rcu_read_unlock();
> release_sock(sk);
> close_fun(sk, timeout);
>
^ permalink raw reply
* [PATCH bpf-next] samples/bpf: fix a compilation failure
From: Yonghong Song @ 2018-09-18 5:08 UTC (permalink / raw)
To: ast, daniel, netdev; +Cc: kernel-team
samples/bpf build failed with the following errors:
$ make samples/bpf/
...
HOSTCC samples/bpf/sockex3_user.o
/data/users/yhs/work/net-next/samples/bpf/sockex3_user.c:16:8: error: redefinition of ‘struct bpf_flow_keys’
struct bpf_flow_keys {
^
In file included from /data/users/yhs/work/net-next/samples/bpf/sockex3_user.c:4:0:
./usr/include/linux/bpf.h:2338:9: note: originally defined here
struct bpf_flow_keys *flow_keys;
^
make[3]: *** [samples/bpf/sockex3_user.o] Error 1
Commit d58e468b1112d ("flow_dissector: implements flow dissector BPF hook")
introduced struct bpf_flow_keys in include/uapi/linux/bpf.h and hence
caused the naming conflict with samples/bpf/sockex3_user.c.
The fix is to rename struct bpf_flow_keys in samples/bpf/sockex3_user.c
to flow_keys to avoid the conflict.
Signed-off-by: Yonghong Song <yhs@fb.com>
---
samples/bpf/sockex3_user.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/samples/bpf/sockex3_user.c b/samples/bpf/sockex3_user.c
index 5ba3ae9d180b..22f74d0e1493 100644
--- a/samples/bpf/sockex3_user.c
+++ b/samples/bpf/sockex3_user.c
@@ -13,7 +13,7 @@
#define PARSE_IP_PROG_FD (prog_fd[0])
#define PROG_ARRAY_FD (map_fd[0])
-struct bpf_flow_keys {
+struct flow_keys {
__be32 src;
__be32 dst;
union {
@@ -64,7 +64,7 @@ int main(int argc, char **argv)
(void) f;
for (i = 0; i < 5; i++) {
- struct bpf_flow_keys key = {}, next_key;
+ struct flow_keys key = {}, next_key;
struct pair value;
sleep(1);
--
2.17.1
^ permalink raw reply related
* [RFC PATCH bpf-next v3 7/7] selftests/bpf: add test cases for queue and stack maps
From: Mauricio Vasquez B @ 2018-09-18 4:53 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, netdev; +Cc: Yonghong Song
In-Reply-To: <153724634652.7866.6354309647800281793.stgit@kernel>
Two types of tests are done:
- test_maps: only userspace api.
- test_progs: userspace api and ebpf helpers.
Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>
---
kernel/bpf/helpers.c | 2
tools/lib/bpf/bpf.c | 12 ++
tools/lib/bpf/bpf.h | 1
tools/testing/selftests/bpf/Makefile | 5 +
tools/testing/selftests/bpf/bpf_helpers.h | 7 +
tools/testing/selftests/bpf/test_maps.c | 130 ++++++++++++++++++++
tools/testing/selftests/bpf/test_progs.c | 99 +++++++++++++++
tools/testing/selftests/bpf/test_queue_map.c | 4 +
tools/testing/selftests/bpf/test_queue_stack_map.h | 59 +++++++++
tools/testing/selftests/bpf/test_stack_map.c | 4 +
10 files changed, 321 insertions(+), 2 deletions(-)
create mode 100644 tools/testing/selftests/bpf/test_queue_map.c
create mode 100644 tools/testing/selftests/bpf/test_queue_stack_map.h
create mode 100644 tools/testing/selftests/bpf/test_stack_map.c
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 5f364e6acaf1..1293cd5240e3 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -76,7 +76,7 @@ const struct bpf_func_proto bpf_map_delete_elem_proto = {
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
-BPF_CALL_4(bpf_map_push_elem, struct bpf_map *, map, void *, value, u32 size,
+BPF_CALL_4(bpf_map_push_elem, struct bpf_map *, map, void *, value, u32, size,
u64, flags)
{
if (map->value_size != size)
diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c
index 60aa4ca8b2c5..7056b2eb554d 100644
--- a/tools/lib/bpf/bpf.c
+++ b/tools/lib/bpf/bpf.c
@@ -286,6 +286,18 @@ int bpf_map_lookup_elem(int fd, const void *key, void *value)
return sys_bpf(BPF_MAP_LOOKUP_ELEM, &attr, sizeof(attr));
}
+int bpf_map_lookup_and_delete_elem(int fd, const void *key, const void *value)
+{
+ union bpf_attr attr;
+
+ bzero(&attr, sizeof(attr));
+ attr.map_fd = fd;
+ attr.key = ptr_to_u64(key);
+ attr.value = ptr_to_u64(value);
+
+ return sys_bpf(BPF_MAP_LOOKUP_AND_DELETE_ELEM, &attr, sizeof(attr));
+}
+
int bpf_map_delete_elem(int fd, const void *key)
{
union bpf_attr attr;
diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h
index 6f38164b2618..6134ed9517d3 100644
--- a/tools/lib/bpf/bpf.h
+++ b/tools/lib/bpf/bpf.h
@@ -86,6 +86,7 @@ int bpf_map_update_elem(int fd, const void *key, const void *value,
__u64 flags);
int bpf_map_lookup_elem(int fd, const void *key, void *value);
+int bpf_map_lookup_and_delete_elem(int fd, const void *key, const void *value);
int bpf_map_delete_elem(int fd, const void *key);
int bpf_map_get_next_key(int fd, const void *key, void *next_key);
int bpf_obj_pin(int fd, const char *pathname);
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index fff7fb1285fc..ad8a2b8fb738 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -35,7 +35,7 @@ TEST_GEN_FILES = test_pkt_access.o test_xdp.o test_l4lb.o test_tcp_estats.o test
test_get_stack_rawtp.o test_sockmap_kern.o test_sockhash_kern.o \
test_lwt_seg6local.o sendmsg4_prog.o sendmsg6_prog.o test_lirc_mode2_kern.o \
get_cgroup_id_kern.o socket_cookie_prog.o test_select_reuseport_kern.o \
- test_skb_cgroup_id_kern.o
+ test_skb_cgroup_id_kern.o test_queue_map.o test_stack_map.o
# Order correspond to 'make run_tests' order
TEST_PROGS := test_kmod.sh \
@@ -110,6 +110,9 @@ CLANG_FLAGS = -I. -I./include/uapi -I../../../include/uapi \
$(OUTPUT)/test_l4lb_noinline.o: CLANG_FLAGS += -fno-inline
$(OUTPUT)/test_xdp_noinline.o: CLANG_FLAGS += -fno-inline
+$(OUTPUT)/test_queue_map.o: test_queue_stack_map.h
+$(OUTPUT)/test_stack_map.o: test_queue_stack_map.h
+
BTF_LLC_PROBE := $(shell $(LLC) -march=bpf -mattr=help 2>&1 | grep dwarfris)
BTF_PAHOLE_PROBE := $(shell $(BTF_PAHOLE) --help 2>&1 | grep BTF)
BTF_OBJCOPY_PROBE := $(shell $(LLVM_OBJCOPY) --help 2>&1 | grep -i 'usage.*llvm')
diff --git a/tools/testing/selftests/bpf/bpf_helpers.h b/tools/testing/selftests/bpf/bpf_helpers.h
index e4be7730222d..bdbe8f84023e 100644
--- a/tools/testing/selftests/bpf/bpf_helpers.h
+++ b/tools/testing/selftests/bpf/bpf_helpers.h
@@ -16,6 +16,13 @@ static int (*bpf_map_update_elem)(void *map, void *key, void *value,
(void *) BPF_FUNC_map_update_elem;
static int (*bpf_map_delete_elem)(void *map, void *key) =
(void *) BPF_FUNC_map_delete_elem;
+static int (*bpf_map_push_elem)(void *map, const void *value, int len,
+ unsigned long long flags) =
+ (void *) BPF_FUNC_map_push_elem;
+static int (*bpf_map_pop_elem)(void *map, void *value, int len) =
+ (void *) BPF_FUNC_map_pop_elem;
+static int (*bpf_map_peek_elem)(void *map, void *value, int len) =
+ (void *) BPF_FUNC_map_peek_elem;
static int (*bpf_probe_read)(void *dst, int size, void *unsafe_ptr) =
(void *) BPF_FUNC_probe_read;
static unsigned long long (*bpf_ktime_get_ns)(void) =
diff --git a/tools/testing/selftests/bpf/test_maps.c b/tools/testing/selftests/bpf/test_maps.c
index 6f54f84144a0..abb21ea731fa 100644
--- a/tools/testing/selftests/bpf/test_maps.c
+++ b/tools/testing/selftests/bpf/test_maps.c
@@ -15,6 +15,7 @@
#include <string.h>
#include <assert.h>
#include <stdlib.h>
+#include <time.h>
#include <sys/wait.h>
#include <sys/socket.h>
@@ -471,6 +472,130 @@ static void test_devmap(int task, void *data)
close(fd);
}
+static void test_queuemap(int task, void *data)
+{
+ const int MAP_SIZE = 32;
+ __u32 vals[MAP_SIZE + MAP_SIZE/2], val;
+ int fd, i;
+
+ /* Fill test values to be used */
+ for (i = 0; i < MAP_SIZE + MAP_SIZE/2; i++)
+ vals[i] = rand();
+
+ /* Invalid key size */
+ fd = bpf_create_map(BPF_MAP_TYPE_QUEUE, 4, sizeof(val), MAP_SIZE,
+ map_flags);
+ assert(fd < 0 && errno == EINVAL);
+
+ /* Invalid value size */
+ fd = bpf_create_map(BPF_MAP_TYPE_QUEUE, 0, 3, MAP_SIZE,map_flags);
+ assert(fd < 0 && errno == EINVAL);
+
+ fd = bpf_create_map(BPF_MAP_TYPE_QUEUE, 0, sizeof(val), MAP_SIZE,
+ map_flags);
+ /* Queue map does not support BPF_F_NO_PREALLOC */
+ if (map_flags & BPF_F_NO_PREALLOC) {
+ assert(fd < 0 && errno == EINVAL);
+ return;
+ }
+ if (fd < 0) {
+ printf("Failed to create queuemap '%s'!\n", strerror(errno));
+ exit(1);
+ }
+
+ /* Push MAP_SIZE elements */
+ for (i = 0; i < MAP_SIZE; i++)
+ assert(bpf_map_update_elem(fd, NULL, &vals[i], 0) == 0);
+
+ /* Check that element cannot be pushed due to max_entries limit */
+ assert(bpf_map_update_elem(fd, NULL, &val, 0) == -1 &&
+ errno == E2BIG);
+
+ /* Peek element */
+ assert(bpf_map_lookup_elem(fd, NULL, &val) == 0 && val == vals[0]);
+
+ /* Replace half elements */
+ for (i = MAP_SIZE; i < MAP_SIZE + MAP_SIZE/2; i++)
+ assert(bpf_map_update_elem(fd, NULL, &vals[i], BPF_EXIST) == 0);
+
+ /* Pop all elements */
+ for (i = MAP_SIZE/2; i < MAP_SIZE + MAP_SIZE/2; i++)
+ assert(bpf_map_lookup_and_delete_elem(fd, NULL, &val) == 0 &&
+ val == vals[i]);
+
+ /* Check that there are not elements left */
+ assert(bpf_map_lookup_and_delete_elem(fd, NULL, &val) == -1 &&
+ errno == ENOENT);
+
+ /* Check that non supported functions set errno to EINVAL */
+ assert(bpf_map_delete_elem(fd, NULL) == -1 && errno == EINVAL);
+ assert(bpf_map_get_next_key(fd, NULL, NULL) == -1 && errno == EINVAL);
+
+ close(fd);
+}
+
+static void test_stackmap(int task, void *data)
+{
+ const int MAP_SIZE = 32;
+ __u32 vals[MAP_SIZE + MAP_SIZE/2], val;
+ int fd, i;
+
+ /* Fill test values to be used */
+ for (i = 0; i < MAP_SIZE + MAP_SIZE/2; i++)
+ vals[i] = rand();
+
+ /* Invalid key size */
+ fd = bpf_create_map(BPF_MAP_TYPE_STACK, 4, sizeof(val), MAP_SIZE,
+ map_flags);
+ assert(fd < 0 && errno == EINVAL);
+
+ /* Invalid value size */
+ fd = bpf_create_map(BPF_MAP_TYPE_STACK, 0, 3, MAP_SIZE,map_flags);
+ assert(fd < 0 && errno == EINVAL);
+
+ fd = bpf_create_map(BPF_MAP_TYPE_STACK, 0, sizeof(val), MAP_SIZE,
+ map_flags);
+ /* Stack map does not support BPF_F_NO_PREALLOC */
+ if (map_flags & BPF_F_NO_PREALLOC) {
+ assert(fd < 0 && errno == EINVAL);
+ return;
+ }
+ if (fd < 0) {
+ printf("Failed to create stackmap '%s'!\n", strerror(errno));
+ exit(1);
+ }
+
+ /* Push MAP_SIZE elements */
+ for (i = 0; i < MAP_SIZE; i++)
+ assert(bpf_map_update_elem(fd, NULL, &vals[i], 0) == 0);
+
+ /* Check that element cannot be pushed due to max_entries limit */
+ assert(bpf_map_update_elem(fd, NULL, &val, 0) == -1 &&
+ errno == E2BIG);
+
+ /* Peek element */
+ assert(bpf_map_lookup_elem(fd, NULL, &val) == 0 && val == vals[i - 1]);
+
+ /* Replace half elements */
+ for (i = MAP_SIZE; i < MAP_SIZE + MAP_SIZE/2; i++)
+ assert(bpf_map_update_elem(fd, NULL, &vals[i], BPF_EXIST) == 0);
+
+ /* Pop all elements */
+ for (i = MAP_SIZE + MAP_SIZE/2 - 1; i >= MAP_SIZE/2; i--)
+ assert(bpf_map_lookup_and_delete_elem(fd, NULL, &val) == 0 &&
+ val == vals[i]);
+
+ /* Check that there are not elements left */
+ assert(bpf_map_lookup_and_delete_elem(fd, NULL, &val) == -1 &&
+ errno == ENOENT);
+
+ /* Check that non supported functions set errno to EINVAL */
+ assert(bpf_map_delete_elem(fd, NULL) == -1 && errno == EINVAL);
+ assert(bpf_map_get_next_key(fd, NULL, NULL) == -1 && errno == EINVAL);
+
+ close(fd);
+}
+
#include <sys/socket.h>
#include <sys/ioctl.h>
#include <arpa/inet.h>
@@ -1430,10 +1555,15 @@ static void run_all_tests(void)
test_map_wronly();
test_reuseport_array();
+
+ test_queuemap(0, NULL);
+ test_stackmap(0, NULL);
}
int main(void)
{
+ srand(time(NULL));
+
map_flags = 0;
run_all_tests();
diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c
index 63a671803ed6..a69b9e1e668d 100644
--- a/tools/testing/selftests/bpf/test_progs.c
+++ b/tools/testing/selftests/bpf/test_progs.c
@@ -1698,8 +1698,105 @@ static void test_task_fd_query_tp(void)
"sys_enter_read");
}
+enum {
+ QUEUE,
+ STACK,
+};
+
+static void test_queue_stack_map(int type)
+{
+ const int MAP_SIZE = 32;
+ __u32 vals[MAP_SIZE], duration, retval, size, val;
+ int i, err, prog_fd, map_in_fd, map_out_fd;
+ char file[32], buf[128];
+ struct bpf_object *obj;
+ struct iphdr *iph = (void *)buf + sizeof(struct ethhdr);
+
+ /* Fill test values to be used */
+ for (i = 0; i < MAP_SIZE; i++)
+ vals[i] = rand();
+
+ if (type == QUEUE)
+ strncpy(file, "./test_queue_map.o", sizeof(file));
+ else if (type == STACK)
+ strncpy(file, "./test_stack_map.o", sizeof(file));
+ else
+ return;
+
+ err = bpf_prog_load(file, BPF_PROG_TYPE_SCHED_CLS, &obj, &prog_fd);
+ if (err) {
+ error_cnt++;
+ return;
+ }
+
+ map_in_fd = bpf_find_map(__func__, obj, "map_in");
+ if (map_in_fd < 0)
+ goto out;
+
+ map_out_fd = bpf_find_map(__func__, obj, "map_out");
+ if (map_out_fd < 0)
+ goto out;
+
+ /* Push 32 elements to the input map */
+ for (i = 0; i < MAP_SIZE; i++) {
+ err = bpf_map_update_elem(map_in_fd, NULL, &vals[i], 0);
+ if (err) {
+ error_cnt++;
+ goto out;
+ }
+ }
+
+ /* The eBPF program pushes iph.saddr in the output map,
+ * pops the input map and saves this value in iph.daddr
+ */
+ for (i = 0; i < MAP_SIZE; i++) {
+ if (type == QUEUE) {
+ val = vals[i];
+ pkt_v4.iph.saddr = vals[i] * 5;
+ } else if (type == STACK) {
+ val = vals[MAP_SIZE - 1 - i];
+ pkt_v4.iph.saddr = vals[MAP_SIZE - 1 - i] * 5;
+ }
+
+ err = bpf_prog_test_run(prog_fd, 1, &pkt_v4, sizeof(pkt_v4),
+ buf, &size, &retval, &duration);
+ if (err || retval || size != sizeof(pkt_v4) ||
+ iph->daddr != val)
+ break;
+ }
+
+ CHECK(err || retval || size != sizeof(pkt_v4) || iph->daddr != val,
+ "bpf_map_pop_elem",
+ "err %d errno %d retval %d size %d iph->daddr %u\n",
+ err, errno, retval, size, iph->daddr);
+
+ /* Queue is empty, program should return TC_ACT_SHOT */
+ err = bpf_prog_test_run(prog_fd, 1, &pkt_v4, sizeof(pkt_v4),
+ buf, &size, &retval, &duration);
+ CHECK(err || retval != 2 /* TC_ACT_SHOT */|| size != sizeof(pkt_v4),
+ "check-queue-stack-map-empty",
+ "err %d errno %d retval %d size %d\n",
+ err, errno, retval, size);
+
+ /* Check that the program pushed elements correctly */
+ for (i = 0; i < MAP_SIZE; i++) {
+ err = bpf_map_lookup_and_delete_elem(map_out_fd, NULL, &val);
+ if (err || val != vals[i] * 5)
+ break;
+ }
+
+ CHECK(i != MAP_SIZE && (err || val != vals[i] * 5),
+ "bpf_map_push_elem", "err %d value %u\n", err, val);
+
+out:
+ pkt_v4.iph.saddr = 0;
+ bpf_object__close(obj);
+}
+
int main(void)
{
+ srand(time(NULL));
+
jit_enabled = is_jit_enabled();
test_pkt_access();
@@ -1719,6 +1816,8 @@ int main(void)
test_get_stack_raw_tp();
test_task_fd_query_rawtp();
test_task_fd_query_tp();
+ test_queue_stack_map(QUEUE);
+ test_queue_stack_map(STACK);
printf("Summary: %d PASSED, %d FAILED\n", pass_cnt, error_cnt);
return error_cnt ? EXIT_FAILURE : EXIT_SUCCESS;
diff --git a/tools/testing/selftests/bpf/test_queue_map.c b/tools/testing/selftests/bpf/test_queue_map.c
new file mode 100644
index 000000000000..3fdb3b9cb038
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_queue_map.c
@@ -0,0 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (c) 2017 Politecnico di Torino
+#define MAP_TYPE BPF_MAP_TYPE_QUEUE
+#include "test_queue_stack_map.h"
diff --git a/tools/testing/selftests/bpf/test_queue_stack_map.h b/tools/testing/selftests/bpf/test_queue_stack_map.h
new file mode 100644
index 000000000000..23a5b5d60069
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_queue_stack_map.h
@@ -0,0 +1,59 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+// Copyright (c) 2017 Politecnico di Torino
+#include <stddef.h>
+#include <string.h>
+#include <linux/bpf.h>
+#include <linux/if_ether.h>
+#include <linux/ip.h>
+#include <linux/pkt_cls.h>
+#include "bpf_helpers.h"
+
+int _version SEC("version") = 1;
+
+struct bpf_map_def __attribute__ ((section("maps"), used)) map_in = {
+ .type = MAP_TYPE,
+ .key_size = 0,
+ .value_size = sizeof(__u32),
+ .max_entries = 32,
+ .map_flags = 0,
+};
+
+struct bpf_map_def __attribute__ ((section("maps"), used)) map_out = {
+ .type = MAP_TYPE,
+ .key_size = 0,
+ .value_size = sizeof(__u32),
+ .max_entries = 32,
+ .map_flags = 0,
+};
+
+SEC("test")
+int _test(struct __sk_buff *skb)
+{
+ void *data_end = (void *)(long)skb->data_end;
+ void *data = (void *)(long)skb->data;
+ struct ethhdr *eth = (struct ethhdr *)(data);
+ __u32 value;
+ int err;
+
+ if (eth + 1 > data_end)
+ return TC_ACT_SHOT;
+
+ struct iphdr *iph = (struct iphdr *)(eth + 1);
+
+ if (iph + 1 > data_end)
+ return TC_ACT_SHOT;
+
+ err = bpf_map_pop_elem(&map_in, &value, sizeof(value));
+ if (err)
+ return TC_ACT_SHOT;
+
+ iph->daddr = value;
+
+ err = bpf_map_push_elem(&map_out, &iph->saddr, sizeof(iph->saddr), 0);
+ if (err)
+ return TC_ACT_SHOT;
+
+ return TC_ACT_OK;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_stack_map.c b/tools/testing/selftests/bpf/test_stack_map.c
new file mode 100644
index 000000000000..5be9159b2927
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_stack_map.c
@@ -0,0 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (c) 2017 Politecnico di Torino
+#define MAP_TYPE BPF_MAP_TYPE_STACK
+#include "test_queue_stack_map.h"
^ permalink raw reply related
* [RFC PATCH bpf-next v3 6/7] Sync uapi/bpf.h to tools/include
From: Mauricio Vasquez B @ 2018-09-18 4:53 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, netdev; +Cc: Yonghong Song
In-Reply-To: <153724634652.7866.6354309647800281793.stgit@kernel>
Sync both files.
Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>
---
tools/include/uapi/linux/bpf.h | 31 ++++++++++++++++++++++++++++++-
1 file changed, 30 insertions(+), 1 deletion(-)
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 66917a4eba27..c899386dcb2b 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -103,6 +103,7 @@ enum bpf_cmd {
BPF_BTF_LOAD,
BPF_BTF_GET_FD_BY_ID,
BPF_TASK_FD_QUERY,
+ BPF_MAP_LOOKUP_AND_DELETE_ELEM,
};
enum bpf_map_type {
@@ -127,6 +128,8 @@ enum bpf_map_type {
BPF_MAP_TYPE_SOCKHASH,
BPF_MAP_TYPE_CGROUP_STORAGE,
BPF_MAP_TYPE_REUSEPORT_SOCKARRAY,
+ BPF_MAP_TYPE_QUEUE,
+ BPF_MAP_TYPE_STACK,
};
enum bpf_prog_type {
@@ -459,6 +462,29 @@ union bpf_attr {
* Return
* 0 on success, or a negative error in case of failure.
*
+ * int bpf_map_push_elem(struct bpf_map *map, const void *value, u32 len,
+ * u64 flags)
+ * Description
+ * Push an element *value* in *map*. *flags* is one of:
+ *
+ * **BPF_EXIST**
+ * If the queue/stack is full, the oldest element is removed to
+ * make room for this.
+ * Return
+ * 0 on success, or a negative error in case of failure.
+ *
+ * int bpf_map_pop_elem(struct bpf_map *map, void *value, u32 len)
+ * Description
+ * Pop an element from *map*.
+ * Return
+ * 0 on success, or a negative error in case of failure.
+ *
+ * int bpf_map_peek_elem(struct bpf_map *map, void *value, u32 len)
+ * Description
+ * Get an element from *map* without removing it.
+ * Return
+ * 0 on success, or a negative error in case of failure.
+ *
* int bpf_probe_read(void *dst, u32 size, const void *src)
* Description
* For tracing programs, safely attempt to read *size* bytes from
@@ -2226,7 +2252,10 @@ union bpf_attr {
FN(get_current_cgroup_id), \
FN(get_local_storage), \
FN(sk_select_reuseport), \
- FN(skb_ancestor_cgroup_id),
+ FN(skb_ancestor_cgroup_id), \
+ FN(map_push_elem), \
+ FN(map_pop_elem), \
+ FN(map_peek_elem),
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
* function eBPF program intends to call
^ permalink raw reply related
* [RFC PATCH bpf-next v3 5/7] bpf: restrict use of peek/push/pop
From: Mauricio Vasquez B @ 2018-09-18 4:52 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, netdev; +Cc: Yonghong Song
In-Reply-To: <153724634652.7866.6354309647800281793.stgit@kernel>
Restrict the use of peek, push and pop helpers only to queue and stack
maps.
Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>
---
kernel/bpf/verifier.c | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index b9e005188f0e..1628ffe48e32 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2084,6 +2084,13 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env,
if (func_id != BPF_FUNC_sk_select_reuseport)
goto error;
break;
+ case BPF_MAP_TYPE_QUEUE:
+ case BPF_MAP_TYPE_STACK:
+ if (func_id != BPF_FUNC_map_peek_elem &&
+ func_id != BPF_FUNC_map_pop_elem &&
+ func_id != BPF_FUNC_map_push_elem)
+ goto error;
+ break;
default:
break;
}
@@ -2139,6 +2146,13 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env,
if (map->map_type != BPF_MAP_TYPE_REUSEPORT_SOCKARRAY)
goto error;
break;
+ case BPF_FUNC_map_peek_elem:
+ case BPF_FUNC_map_pop_elem:
+ case BPF_FUNC_map_push_elem:
+ if (map->map_type != BPF_MAP_TYPE_QUEUE &&
+ map->map_type != BPF_MAP_TYPE_STACK)
+ goto error;
+ break;
default:
break;
}
^ permalink raw reply related
* [RFC PATCH bpf-next v3 4/7] bpf: add bpf queue and stack maps
From: Mauricio Vasquez B @ 2018-09-18 4:52 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, netdev; +Cc: Yonghong Song
In-Reply-To: <153724634652.7866.6354309647800281793.stgit@kernel>
Implement two new kind of maps that support the peek, push and pop
operations.
A use case for this is to keep track of a pool of elements, like
network ports in a SNAT.
Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>
---
include/linux/bpf.h | 3
include/linux/bpf_types.h | 2
include/uapi/linux/bpf.h | 30 ++++
kernel/bpf/Makefile | 2
kernel/bpf/core.c | 3
kernel/bpf/helpers.c | 98 ++++++++++++++
kernel/bpf/queue_stack_maps.c | 291 +++++++++++++++++++++++++++++++++++++++++
kernel/bpf/verifier.c | 5 +
net/core/filter.c | 6 +
9 files changed, 437 insertions(+), 3 deletions(-)
create mode 100644 kernel/bpf/queue_stack_maps.c
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index c63a44381d3f..8e924b5c5a0e 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -807,6 +807,9 @@ static inline int bpf_fd_reuseport_array_update_elem(struct bpf_map *map,
extern const struct bpf_func_proto bpf_map_lookup_elem_proto;
extern const struct bpf_func_proto bpf_map_update_elem_proto;
extern const struct bpf_func_proto bpf_map_delete_elem_proto;
+extern const struct bpf_func_proto bpf_map_push_elem_proto;
+extern const struct bpf_func_proto bpf_map_pop_elem_proto;
+extern const struct bpf_func_proto bpf_map_peek_elem_proto;
extern const struct bpf_func_proto bpf_get_prandom_u32_proto;
extern const struct bpf_func_proto bpf_get_smp_processor_id_proto;
diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h
index 33f7f574b983..903a446f14c3 100644
--- a/include/linux/bpf_types.h
+++ b/include/linux/bpf_types.h
@@ -67,3 +67,5 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_XSKMAP, xsk_map_ops)
BPF_MAP_TYPE(BPF_MAP_TYPE_REUSEPORT_SOCKARRAY, reuseport_array_ops)
#endif
#endif
+BPF_MAP_TYPE(BPF_MAP_TYPE_QUEUE, queue_map_ops)
+BPF_MAP_TYPE(BPF_MAP_TYPE_STACK, stack_map_ops)
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 4cda584c6640..c899386dcb2b 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -128,6 +128,8 @@ enum bpf_map_type {
BPF_MAP_TYPE_SOCKHASH,
BPF_MAP_TYPE_CGROUP_STORAGE,
BPF_MAP_TYPE_REUSEPORT_SOCKARRAY,
+ BPF_MAP_TYPE_QUEUE,
+ BPF_MAP_TYPE_STACK,
};
enum bpf_prog_type {
@@ -460,6 +462,29 @@ union bpf_attr {
* Return
* 0 on success, or a negative error in case of failure.
*
+ * int bpf_map_push_elem(struct bpf_map *map, const void *value, u32 len,
+ * u64 flags)
+ * Description
+ * Push an element *value* in *map*. *flags* is one of:
+ *
+ * **BPF_EXIST**
+ * If the queue/stack is full, the oldest element is removed to
+ * make room for this.
+ * Return
+ * 0 on success, or a negative error in case of failure.
+ *
+ * int bpf_map_pop_elem(struct bpf_map *map, void *value, u32 len)
+ * Description
+ * Pop an element from *map*.
+ * Return
+ * 0 on success, or a negative error in case of failure.
+ *
+ * int bpf_map_peek_elem(struct bpf_map *map, void *value, u32 len)
+ * Description
+ * Get an element from *map* without removing it.
+ * Return
+ * 0 on success, or a negative error in case of failure.
+ *
* int bpf_probe_read(void *dst, u32 size, const void *src)
* Description
* For tracing programs, safely attempt to read *size* bytes from
@@ -2227,7 +2252,10 @@ union bpf_attr {
FN(get_current_cgroup_id), \
FN(get_local_storage), \
FN(sk_select_reuseport), \
- FN(skb_ancestor_cgroup_id),
+ FN(skb_ancestor_cgroup_id), \
+ FN(map_push_elem), \
+ FN(map_pop_elem), \
+ FN(map_peek_elem),
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
* function eBPF program intends to call
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index e656bce87c8f..2d77bc5b2aca 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -3,7 +3,7 @@ obj-y := core.o
obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o
obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o
-obj-$(CONFIG_BPF_SYSCALL) += local_storage.o
+obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o
obj-$(CONFIG_BPF_SYSCALL) += disasm.o
obj-$(CONFIG_BPF_SYSCALL) += btf.o
ifeq ($(CONFIG_NET),y)
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 3f5bf1af0826..8d2db076d123 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1783,6 +1783,9 @@ BPF_CALL_0(bpf_user_rnd_u32)
const struct bpf_func_proto bpf_map_lookup_elem_proto __weak;
const struct bpf_func_proto bpf_map_update_elem_proto __weak;
const struct bpf_func_proto bpf_map_delete_elem_proto __weak;
+const struct bpf_func_proto bpf_map_push_elem_proto __weak;
+const struct bpf_func_proto bpf_map_pop_elem_proto __weak;
+const struct bpf_func_proto bpf_map_peek_elem_proto __weak;
const struct bpf_func_proto bpf_get_prandom_u32_proto __weak;
const struct bpf_func_proto bpf_get_smp_processor_id_proto __weak;
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 1991466b8327..5f364e6acaf1 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -76,6 +76,104 @@ const struct bpf_func_proto bpf_map_delete_elem_proto = {
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
+BPF_CALL_4(bpf_map_push_elem, struct bpf_map *, map, void *, value, u32 size,
+ u64, flags)
+{
+ if (map->value_size != size)
+ return -EINVAL;
+
+ return map->ops->map_update_elem(map, NULL, value, flags);
+}
+
+const struct bpf_func_proto bpf_map_push_elem_proto = {
+ .func = bpf_map_push_elem,
+ .gpl_only = false,
+ .pkt_access = true,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_CONST_MAP_PTR,
+ .arg2_type = ARG_PTR_TO_MEM,
+ .arg3_type = ARG_CONST_SIZE,
+ .arg4_type = ARG_ANYTHING,
+};
+
+BPF_CALL_3(bpf_map_pop_elem, struct bpf_map *, map, void *, value, u32, size)
+{
+ void *ptr;
+
+ if (map->value_size != size)
+ return -EINVAL;
+
+ ptr = map->ops->map_lookup_and_delete_elem(map, NULL);
+ if (!ptr)
+ return -ENOENT;
+
+ switch (size) {
+ case 1:
+ *(u8 *) value = *(u8 *) ptr;
+ break;
+ case 2:
+ *(u16 *) value = *(u16 *) ptr;
+ break;
+ case 4:
+ *(u32 *) value = *(u32 *) ptr;
+ break;
+ case 8:
+ *(u64 *) value = *(u64 *) ptr;
+ break;
+ }
+
+ return 0;
+}
+
+const struct bpf_func_proto bpf_map_pop_elem_proto = {
+ .func = bpf_map_pop_elem,
+ .gpl_only = false,
+ .pkt_access = true,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_CONST_MAP_PTR,
+ .arg2_type = ARG_PTR_TO_UNINIT_MEM,
+ .arg3_type = ARG_CONST_SIZE,
+};
+
+BPF_CALL_3(bpf_map_peek_elem, struct bpf_map *, map, void *, value, u32, size)
+{
+ void *ptr;
+
+ if (map->value_size != size)
+ return -EINVAL;
+
+ ptr = map->ops->map_lookup_elem(map, NULL);
+ if (!ptr)
+ return -ENOENT;
+
+ switch (size) {
+ case 1:
+ *(u8 *) value = *(u8 *) ptr;
+ break;
+ case 2:
+ *(u16 *) value = *(u16 *) ptr;
+ break;
+ case 4:
+ *(u32 *) value = *(u32 *) ptr;
+ break;
+ case 8:
+ *(u64 *) value = *(u64 *) ptr;
+ break;
+ }
+
+ return 0;
+}
+
+const struct bpf_func_proto bpf_map_peek_elem_proto = {
+ .func = bpf_map_pop_elem,
+ .gpl_only = false,
+ .pkt_access = true,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_CONST_MAP_PTR,
+ .arg2_type = ARG_PTR_TO_UNINIT_MEM,
+ .arg3_type = ARG_CONST_SIZE,
+};
+
const struct bpf_func_proto bpf_get_prandom_u32_proto = {
.func = bpf_user_rnd_u32,
.gpl_only = false,
diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c
new file mode 100644
index 000000000000..10c081f3f02b
--- /dev/null
+++ b/kernel/bpf/queue_stack_maps.c
@@ -0,0 +1,291 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * queue_stack_maps.c: BPF queue and stack maps
+ *
+ * Copyright (c) 2018 Politecnico di Torino
+ */
+#include <linux/bpf.h>
+#include <linux/list.h>
+#include <linux/slab.h>
+#include "percpu_freelist.h"
+
+#define QUEUE_STACK_CREATE_FLAG_MASK \
+ (BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY)
+
+
+struct bpf_queue_stack {
+ struct bpf_map map;
+ raw_spinlock_t lock;
+ u32 head, tail;
+ u32 index_mask;
+ u32 count;
+
+ char elements[0] __aligned(8);
+};
+
+static struct bpf_queue_stack *bpf_queue_stack(struct bpf_map *map)
+{
+ return container_of(map, struct bpf_queue_stack, map);
+}
+
+static bool queue_stack_map_is_empty(struct bpf_queue_stack *qs)
+{
+ return qs->count == 0;
+}
+
+static bool queue_stack_map_is_full(struct bpf_queue_stack *qs)
+{
+ return qs->count == qs->map.max_entries;
+}
+
+/* Called from syscall */
+static int queue_stack_map_alloc_check(union bpf_attr *attr)
+{
+ /* check sanity of attributes */
+ if (attr->max_entries == 0 || attr->key_size != 0 ||
+ (attr->value_size != 1 && attr->value_size != 2 &&
+ attr->value_size != 4 && attr->value_size != 8) ||
+ attr->map_flags & ~QUEUE_STACK_CREATE_FLAG_MASK)
+ return -EINVAL;
+
+ if (attr->value_size > KMALLOC_MAX_SIZE)
+ /* if value_size is bigger, the user space won't be able to
+ * access the elements.
+ */
+ return -E2BIG;
+
+ return 0;
+}
+
+static struct bpf_map *queue_stack_map_alloc(union bpf_attr *attr)
+{
+ int ret, numa_node = bpf_map_attr_numa_node(attr);
+ u32 max_entries, value_size, index_mask;
+ u64 queue_size, cost, mask64;
+ struct bpf_queue_stack *qs;
+
+ max_entries = attr->max_entries;
+ value_size = attr->value_size;
+
+ /* From arraymap.c:
+ * On 32 bit archs roundup_pow_of_two() with max_entries that has
+ * upper most bit set in u32 space is undefined behavior due to
+ * resulting 1U << 32, so do it manually here in u64 space.
+ */
+ mask64 = fls_long(max_entries - 1);
+ mask64 = 1ULL << mask64;
+ mask64 -= 1;
+
+ index_mask = mask64;
+
+ /* Round up queue size to nearest power of 2 */
+ max_entries = index_mask + 1;
+ /* Check for overflows. */
+ if (max_entries < attr->max_entries)
+ return ERR_PTR(-E2BIG);
+
+ queue_size = sizeof(*qs) + (u64) value_size * max_entries;
+
+ cost = queue_size;
+ if (cost >= U32_MAX - PAGE_SIZE)
+ return ERR_PTR(-E2BIG);
+
+ cost = round_up(cost, PAGE_SIZE) >> PAGE_SHIFT;
+
+ ret = bpf_map_precharge_memlock(cost);
+ if (ret < 0)
+ return ERR_PTR(ret);
+
+ qs = bpf_map_area_alloc(queue_size, numa_node);
+ if (!qs)
+ return ERR_PTR(-ENOMEM);
+
+ memset(qs, 0, sizeof(*qs));
+
+ bpf_map_init_from_attr(&qs->map, attr);
+
+ qs->map.pages = cost;
+ qs->index_mask = index_mask;
+
+ raw_spin_lock_init(&qs->lock);
+
+ return &qs->map;
+}
+
+/* Called when map->refcnt goes to zero, either from workqueue or from syscall */
+static void queue_stack_map_free(struct bpf_map *map)
+{
+ struct bpf_queue_stack *qs = bpf_queue_stack(map);
+
+ /* at this point bpf_prog->aux->refcnt == 0 and this map->refcnt == 0,
+ * so the programs (can be more than one that used this map) were
+ * disconnected from events. Wait for outstanding critical sections in
+ * these programs to complete
+ */
+ synchronize_rcu();
+
+ bpf_map_area_free(qs);
+}
+
+static void *__queue_map_lookup(struct bpf_map *map, bool delete)
+{
+ struct bpf_queue_stack *qs = bpf_queue_stack(map);
+ unsigned long flags;
+ void *ptr = NULL;
+
+ raw_spin_lock_irqsave(&qs->lock, flags);
+
+ if (queue_stack_map_is_empty(qs))
+ goto out;
+
+ ptr = &qs->elements[qs->tail * qs->map.value_size];
+
+ if (delete) {
+ qs->tail = (qs->tail + 1) & qs->index_mask;
+ qs->count--;
+ }
+
+out:
+ raw_spin_unlock_irqrestore(&qs->lock, flags);
+ return ptr;
+}
+
+
+static void *__stack_map_lookup(struct bpf_map *map, bool delete)
+{
+ struct bpf_queue_stack *qs = bpf_queue_stack(map);
+ unsigned long flags;
+ void *ptr = NULL;
+ u32 index;
+
+ raw_spin_lock_irqsave(&qs->lock, flags);
+
+ if (queue_stack_map_is_empty(qs))
+ goto out;
+
+ index = (qs->head - 1) & qs->index_mask;
+ ptr = &qs->elements[index * qs->map.value_size];
+
+ if (delete) {
+ qs->head = (qs->head - 1) & qs->index_mask;
+ qs->count--;
+ }
+
+out:
+ raw_spin_unlock_irqrestore(&qs->lock, flags);
+ return ptr;
+}
+
+/* Called from syscall or from eBPF program */
+static void *queue_map_lookup_elem(struct bpf_map *map, void *key)
+{
+ return __queue_map_lookup(map, false);
+}
+
+/* Called from syscall or from eBPF program */
+static void *stack_map_lookup_elem(struct bpf_map *map, void *key)
+{
+ return __stack_map_lookup(map, false);
+}
+
+/* Called from syscall or from eBPF program */
+static void *queue_map_lookup_and_delete_elem(struct bpf_map *map, void *key)
+{
+ return __queue_map_lookup(map, true);
+}
+
+/* Called from syscall or from eBPF program */
+static void *stack_map_lookup_and_delete_elem(struct bpf_map *map, void *key)
+{
+ return __stack_map_lookup(map, true);
+}
+
+/* Called from syscall or from eBPF program */
+static int queue_stack_map_update_elem(struct bpf_map *map, void *key,
+ void *value, u64 flags)
+{
+ struct bpf_queue_stack *qs = bpf_queue_stack(map);
+ unsigned long irq_flags;
+ int err = 0;
+ void *dst;
+
+ /* BPF_EXIST is used to force making room for a new element in case the
+ * map is full
+ */
+ bool replace = (flags & BPF_EXIST);
+
+ /* Check supported flags for queue and stqueue_map_is_preallocack maps */
+ if (flags & BPF_NOEXIST || flags > BPF_EXIST)
+ return -EINVAL;
+
+ raw_spin_lock_irqsave(&qs->lock, irq_flags);
+
+ if (queue_stack_map_is_full(qs)) {
+ if (!replace) {
+ err = -E2BIG;
+ goto out;
+ }
+ /* advance tail pointer to overwrite oldest element */
+ qs->tail = (qs->tail + 1) & qs->index_mask;
+ qs->count--;
+ }
+
+ dst = &qs->elements[qs->head * qs->map.value_size];
+
+ switch (qs->map.value_size) {
+ case 1:
+ *(u8 *) dst = *(u8 *) value;
+ break;
+ case 2:
+ *(u16 *) dst = *(u16 *) value;
+ break;
+ case 4:
+ *(u32 *) dst = *(u32 *) value;
+ break;
+ case 8:
+ *(u64 *) dst = *(u64 *) value;
+ break;
+ }
+
+ qs->head = (qs->head + 1) & qs->index_mask;
+ qs->count++;
+
+out:
+ raw_spin_unlock_irqrestore(&qs->lock, irq_flags);
+ return err;
+}
+
+/* Called from syscall or from eBPF program */
+static int queue_stack_map_delete_elem(struct bpf_map *map, void *key)
+{
+ return -EINVAL;
+}
+
+/* Called from syscall */
+static int queue_stack_map_get_next_key(struct bpf_map *map, void *key,
+ void *next_key)
+{
+ return -EINVAL;
+}
+
+const struct bpf_map_ops queue_map_ops = {
+ .map_alloc_check = queue_stack_map_alloc_check,
+ .map_alloc = queue_stack_map_alloc,
+ .map_free = queue_stack_map_free,
+ .map_lookup_elem = queue_map_lookup_elem,
+ .map_lookup_and_delete_elem = queue_map_lookup_and_delete_elem,
+ .map_update_elem = queue_stack_map_update_elem,
+ .map_delete_elem = queue_stack_map_delete_elem,
+ .map_get_next_key = queue_stack_map_get_next_key,
+};
+
+const struct bpf_map_ops stack_map_ops = {
+ .map_alloc_check = queue_stack_map_alloc_check,
+ .map_alloc = queue_stack_map_alloc,
+ .map_free = queue_stack_map_free,
+ .map_lookup_elem = stack_map_lookup_elem,
+ .map_lookup_and_delete_elem = stack_map_lookup_and_delete_elem,
+ .map_update_elem = queue_stack_map_update_elem,
+ .map_delete_elem = queue_stack_map_delete_elem,
+ .map_get_next_key = queue_stack_map_get_next_key,
+};
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index f4ff0c569e54..b9e005188f0e 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2369,7 +2369,10 @@ record_func_map(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
if (func_id != BPF_FUNC_tail_call &&
func_id != BPF_FUNC_map_lookup_elem &&
func_id != BPF_FUNC_map_update_elem &&
- func_id != BPF_FUNC_map_delete_elem)
+ func_id != BPF_FUNC_map_delete_elem &&
+ func_id != BPF_FUNC_map_push_elem &&
+ func_id != BPF_FUNC_map_pop_elem &&
+ func_id != BPF_FUNC_map_peek_elem)
return 0;
if (meta->map_ptr == NULL) {
diff --git a/net/core/filter.c b/net/core/filter.c
index feb578506009..c7b73376c23a 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -4839,6 +4839,12 @@ bpf_base_func_proto(enum bpf_func_id func_id)
return &bpf_map_update_elem_proto;
case BPF_FUNC_map_delete_elem:
return &bpf_map_delete_elem_proto;
+ case BPF_FUNC_map_push_elem:
+ return &bpf_map_push_elem_proto;
+ case BPF_FUNC_map_pop_elem:
+ return &bpf_map_pop_elem_proto;
+ case BPF_FUNC_map_peek_elem:
+ return &bpf_map_peek_elem_proto;
case BPF_FUNC_get_prandom_u32:
return &bpf_get_prandom_u32_proto;
case BPF_FUNC_get_smp_processor_id:
^ permalink raw reply related
* [RFC PATCH bpf-next v3 3/7] bpf: add lookup_and_delete map operation
From: Mauricio Vasquez B @ 2018-09-18 4:52 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, netdev; +Cc: Yonghong Song
In-Reply-To: <153724634652.7866.6354309647800281793.stgit@kernel>
The following patch implements a bpf queue/stack maps that
provides the peek/pop/push functions. There is not a direct
relationship between those functions and the current operations
supported by a map, hence a new lookup_and_delete map operation
is added, this operation would be used by the pop helper.
A pop operation is not added because it will too specific to
stack/queue maps, instead this new operation could be useful
for other maps as well.
Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>
---
include/linux/bpf.h | 1 +
include/uapi/linux/bpf.h | 1 +
kernel/bpf/syscall.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 84 insertions(+)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 523481a3471b..c63a44381d3f 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -39,6 +39,7 @@ struct bpf_map_ops {
void *(*map_lookup_elem)(struct bpf_map *map, void *key);
int (*map_update_elem)(struct bpf_map *map, void *key, void *value, u64 flags);
int (*map_delete_elem)(struct bpf_map *map, void *key);
+ void *(*map_lookup_and_delete_elem)(struct bpf_map *map, void *key);
/* funcs called by prog_array and perf_event_array map */
void *(*map_fd_get_ptr)(struct bpf_map *map, struct file *map_file,
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 66917a4eba27..4cda584c6640 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -103,6 +103,7 @@ enum bpf_cmd {
BPF_BTF_LOAD,
BPF_BTF_GET_FD_BY_ID,
BPF_TASK_FD_QUERY,
+ BPF_MAP_LOOKUP_AND_DELETE_ELEM,
};
enum bpf_map_type {
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index f2d4e4f280dc..7d429123a298 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -968,6 +968,85 @@ static int map_get_next_key(union bpf_attr *attr)
return err;
}
+#define BPF_MAP_LOOKUP_AND_DELETE_ELEM_LAST_FIELD value
+
+static int map_lookup_and_delete_elem(union bpf_attr *attr)
+{
+ void __user *ukey = u64_to_user_ptr(attr->key);
+ void __user *uvalue = u64_to_user_ptr(attr->value);
+ int ufd = attr->map_fd;
+ struct bpf_map *map;
+ void *key, *value, *ptr;
+ u32 value_size;
+ struct fd f;
+ int err;
+
+ if (CHECK_ATTR(BPF_MAP_LOOKUP_ELEM))
+ return -EINVAL;
+
+ f = fdget(ufd);
+ map = __bpf_map_get(f);
+ if (IS_ERR(map))
+ return PTR_ERR(map);
+
+ if (!(f.file->f_mode & FMODE_CAN_WRITE)) {
+ err = -EPERM;
+ goto err_put;
+ }
+
+ if (!map->ops->map_lookup_and_delete_elem) {
+ err = -ENOTSUPP;
+ goto err_put;
+ }
+
+ key = __bpf_copy_key(ukey, map->key_size);
+ if (IS_ERR(key)) {
+ err = PTR_ERR(key);
+ goto err_put;
+ }
+
+ value_size = map->value_size;
+
+ err = -ENOMEM;
+ value = kmalloc(value_size, GFP_USER | __GFP_NOWARN);
+ if (!value)
+ goto free_key;
+
+ err = -EFAULT;
+ if (copy_from_user(value, uvalue, value_size) != 0)
+ goto free_value;
+
+ /* must increment bpf_prog_active to avoid kprobe+bpf triggering from
+ * inside bpf map update or delete otherwise deadlocks are possible
+ */
+ preempt_disable();
+ __this_cpu_inc(bpf_prog_active);
+ rcu_read_lock();
+ ptr = map->ops->map_lookup_and_delete_elem(map, key);
+ if (ptr)
+ memcpy(value, ptr, value_size);
+ rcu_read_unlock();
+ err = ptr ? 0 : -ENOENT;
+ __this_cpu_dec(bpf_prog_active);
+ preempt_enable();
+
+ if (err)
+ goto free_value;
+
+ if (copy_to_user(uvalue, value, value_size) != 0)
+ goto free_value;
+
+ err = 0;
+
+free_value:
+ kfree(value);
+free_key:
+ kfree(key);
+err_put:
+ fdput(f);
+ return err;
+}
+
static const struct bpf_prog_ops * const bpf_prog_types[] = {
#define BPF_PROG_TYPE(_id, _name) \
[_id] = & _name ## _prog_ops,
@@ -2428,6 +2507,9 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz
case BPF_TASK_FD_QUERY:
err = bpf_task_fd_query(&attr, uattr);
break;
+ case BPF_MAP_LOOKUP_AND_DELETE_ELEM:
+ err = map_lookup_and_delete_elem(&attr);
+ break;
default:
err = -EINVAL;
break;
^ permalink raw reply related
* [RFC PATCH bpf-next v3 1/7] bpf: rename stack trace map
From: Mauricio Vasquez B @ 2018-09-18 4:52 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, netdev; +Cc: Yonghong Song
In-Reply-To: <153724634652.7866.6354309647800281793.stgit@kernel>
In the following patches queue and stack maps (FIFO and LIFO
datastructures) will be implemented. In order to avoid confusion and
a possible name clash rename stackmap.c to stacktracemap.c and
stack_map_ops to stack_trace_map_ops
Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>
---
include/linux/bpf_types.h | 2
kernel/bpf/Makefile | 2
kernel/bpf/stackmap.c | 624 --------------------------------------------
kernel/bpf/stacktracemap.c | 624 ++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 626 insertions(+), 626 deletions(-)
delete mode 100644 kernel/bpf/stackmap.c
create mode 100644 kernel/bpf/stacktracemap.c
diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h
index cd26c090e7c0..33f7f574b983 100644
--- a/include/linux/bpf_types.h
+++ b/include/linux/bpf_types.h
@@ -49,7 +49,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_LRU_HASH, htab_lru_map_ops)
BPF_MAP_TYPE(BPF_MAP_TYPE_LRU_PERCPU_HASH, htab_lru_percpu_map_ops)
BPF_MAP_TYPE(BPF_MAP_TYPE_LPM_TRIE, trie_map_ops)
#ifdef CONFIG_PERF_EVENTS
-BPF_MAP_TYPE(BPF_MAP_TYPE_STACK_TRACE, stack_map_ops)
+BPF_MAP_TYPE(BPF_MAP_TYPE_STACK_TRACE, stack_trace_map_ops)
#endif
BPF_MAP_TYPE(BPF_MAP_TYPE_ARRAY_OF_MAPS, array_of_maps_map_ops)
BPF_MAP_TYPE(BPF_MAP_TYPE_HASH_OF_MAPS, htab_of_maps_map_ops)
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 0488b8258321..e656bce87c8f 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -20,7 +20,7 @@ endif
endif
endif
ifeq ($(CONFIG_PERF_EVENTS),y)
-obj-$(CONFIG_BPF_SYSCALL) += stackmap.o
+obj-$(CONFIG_BPF_SYSCALL) += stacktracemap.o
endif
obj-$(CONFIG_CGROUP_BPF) += cgroup.o
ifeq ($(CONFIG_INET),y)
diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c
deleted file mode 100644
index 8061a439ef18..000000000000
--- a/kernel/bpf/stackmap.c
+++ /dev/null
@@ -1,624 +0,0 @@
-/* Copyright (c) 2016 Facebook
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of version 2 of the GNU General Public
- * License as published by the Free Software Foundation.
- */
-#include <linux/bpf.h>
-#include <linux/jhash.h>
-#include <linux/filter.h>
-#include <linux/stacktrace.h>
-#include <linux/perf_event.h>
-#include <linux/elf.h>
-#include <linux/pagemap.h>
-#include <linux/irq_work.h>
-#include "percpu_freelist.h"
-
-#define STACK_CREATE_FLAG_MASK \
- (BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY | \
- BPF_F_STACK_BUILD_ID)
-
-struct stack_map_bucket {
- struct pcpu_freelist_node fnode;
- u32 hash;
- u32 nr;
- u64 data[];
-};
-
-struct bpf_stack_map {
- struct bpf_map map;
- void *elems;
- struct pcpu_freelist freelist;
- u32 n_buckets;
- struct stack_map_bucket *buckets[];
-};
-
-/* irq_work to run up_read() for build_id lookup in nmi context */
-struct stack_map_irq_work {
- struct irq_work irq_work;
- struct rw_semaphore *sem;
-};
-
-static void do_up_read(struct irq_work *entry)
-{
- struct stack_map_irq_work *work;
-
- work = container_of(entry, struct stack_map_irq_work, irq_work);
- up_read(work->sem);
- work->sem = NULL;
-}
-
-static DEFINE_PER_CPU(struct stack_map_irq_work, up_read_work);
-
-static inline bool stack_map_use_build_id(struct bpf_map *map)
-{
- return (map->map_flags & BPF_F_STACK_BUILD_ID);
-}
-
-static inline int stack_map_data_size(struct bpf_map *map)
-{
- return stack_map_use_build_id(map) ?
- sizeof(struct bpf_stack_build_id) : sizeof(u64);
-}
-
-static int prealloc_elems_and_freelist(struct bpf_stack_map *smap)
-{
- u32 elem_size = sizeof(struct stack_map_bucket) + smap->map.value_size;
- int err;
-
- smap->elems = bpf_map_area_alloc(elem_size * smap->map.max_entries,
- smap->map.numa_node);
- if (!smap->elems)
- return -ENOMEM;
-
- err = pcpu_freelist_init(&smap->freelist);
- if (err)
- goto free_elems;
-
- pcpu_freelist_populate(&smap->freelist, smap->elems, elem_size,
- smap->map.max_entries);
- return 0;
-
-free_elems:
- bpf_map_area_free(smap->elems);
- return err;
-}
-
-/* Called from syscall */
-static struct bpf_map *stack_map_alloc(union bpf_attr *attr)
-{
- u32 value_size = attr->value_size;
- struct bpf_stack_map *smap;
- u64 cost, n_buckets;
- int err;
-
- if (!capable(CAP_SYS_ADMIN))
- return ERR_PTR(-EPERM);
-
- if (attr->map_flags & ~STACK_CREATE_FLAG_MASK)
- return ERR_PTR(-EINVAL);
-
- /* check sanity of attributes */
- if (attr->max_entries == 0 || attr->key_size != 4 ||
- value_size < 8 || value_size % 8)
- return ERR_PTR(-EINVAL);
-
- BUILD_BUG_ON(sizeof(struct bpf_stack_build_id) % sizeof(u64));
- if (attr->map_flags & BPF_F_STACK_BUILD_ID) {
- if (value_size % sizeof(struct bpf_stack_build_id) ||
- value_size / sizeof(struct bpf_stack_build_id)
- > sysctl_perf_event_max_stack)
- return ERR_PTR(-EINVAL);
- } else if (value_size / 8 > sysctl_perf_event_max_stack)
- return ERR_PTR(-EINVAL);
-
- /* hash table size must be power of 2 */
- n_buckets = roundup_pow_of_two(attr->max_entries);
-
- cost = n_buckets * sizeof(struct stack_map_bucket *) + sizeof(*smap);
- if (cost >= U32_MAX - PAGE_SIZE)
- return ERR_PTR(-E2BIG);
-
- smap = bpf_map_area_alloc(cost, bpf_map_attr_numa_node(attr));
- if (!smap)
- return ERR_PTR(-ENOMEM);
-
- err = -E2BIG;
- cost += n_buckets * (value_size + sizeof(struct stack_map_bucket));
- if (cost >= U32_MAX - PAGE_SIZE)
- goto free_smap;
-
- bpf_map_init_from_attr(&smap->map, attr);
- smap->map.value_size = value_size;
- smap->n_buckets = n_buckets;
- smap->map.pages = round_up(cost, PAGE_SIZE) >> PAGE_SHIFT;
-
- err = bpf_map_precharge_memlock(smap->map.pages);
- if (err)
- goto free_smap;
-
- err = get_callchain_buffers(sysctl_perf_event_max_stack);
- if (err)
- goto free_smap;
-
- err = prealloc_elems_and_freelist(smap);
- if (err)
- goto put_buffers;
-
- return &smap->map;
-
-put_buffers:
- put_callchain_buffers();
-free_smap:
- bpf_map_area_free(smap);
- return ERR_PTR(err);
-}
-
-#define BPF_BUILD_ID 3
-/*
- * Parse build id from the note segment. This logic can be shared between
- * 32-bit and 64-bit system, because Elf32_Nhdr and Elf64_Nhdr are
- * identical.
- */
-static inline int stack_map_parse_build_id(void *page_addr,
- unsigned char *build_id,
- void *note_start,
- Elf32_Word note_size)
-{
- Elf32_Word note_offs = 0, new_offs;
-
- /* check for overflow */
- if (note_start < page_addr || note_start + note_size < note_start)
- return -EINVAL;
-
- /* only supports note that fits in the first page */
- if (note_start + note_size > page_addr + PAGE_SIZE)
- return -EINVAL;
-
- while (note_offs + sizeof(Elf32_Nhdr) < note_size) {
- Elf32_Nhdr *nhdr = (Elf32_Nhdr *)(note_start + note_offs);
-
- if (nhdr->n_type == BPF_BUILD_ID &&
- nhdr->n_namesz == sizeof("GNU") &&
- nhdr->n_descsz == BPF_BUILD_ID_SIZE) {
- memcpy(build_id,
- note_start + note_offs +
- ALIGN(sizeof("GNU"), 4) + sizeof(Elf32_Nhdr),
- BPF_BUILD_ID_SIZE);
- return 0;
- }
- new_offs = note_offs + sizeof(Elf32_Nhdr) +
- ALIGN(nhdr->n_namesz, 4) + ALIGN(nhdr->n_descsz, 4);
- if (new_offs <= note_offs) /* overflow */
- break;
- note_offs = new_offs;
- }
- return -EINVAL;
-}
-
-/* Parse build ID from 32-bit ELF */
-static int stack_map_get_build_id_32(void *page_addr,
- unsigned char *build_id)
-{
- Elf32_Ehdr *ehdr = (Elf32_Ehdr *)page_addr;
- Elf32_Phdr *phdr;
- int i;
-
- /* only supports phdr that fits in one page */
- if (ehdr->e_phnum >
- (PAGE_SIZE - sizeof(Elf32_Ehdr)) / sizeof(Elf32_Phdr))
- return -EINVAL;
-
- phdr = (Elf32_Phdr *)(page_addr + sizeof(Elf32_Ehdr));
-
- for (i = 0; i < ehdr->e_phnum; ++i)
- if (phdr[i].p_type == PT_NOTE)
- return stack_map_parse_build_id(page_addr, build_id,
- page_addr + phdr[i].p_offset,
- phdr[i].p_filesz);
- return -EINVAL;
-}
-
-/* Parse build ID from 64-bit ELF */
-static int stack_map_get_build_id_64(void *page_addr,
- unsigned char *build_id)
-{
- Elf64_Ehdr *ehdr = (Elf64_Ehdr *)page_addr;
- Elf64_Phdr *phdr;
- int i;
-
- /* only supports phdr that fits in one page */
- if (ehdr->e_phnum >
- (PAGE_SIZE - sizeof(Elf64_Ehdr)) / sizeof(Elf64_Phdr))
- return -EINVAL;
-
- phdr = (Elf64_Phdr *)(page_addr + sizeof(Elf64_Ehdr));
-
- for (i = 0; i < ehdr->e_phnum; ++i)
- if (phdr[i].p_type == PT_NOTE)
- return stack_map_parse_build_id(page_addr, build_id,
- page_addr + phdr[i].p_offset,
- phdr[i].p_filesz);
- return -EINVAL;
-}
-
-/* Parse build ID of ELF file mapped to vma */
-static int stack_map_get_build_id(struct vm_area_struct *vma,
- unsigned char *build_id)
-{
- Elf32_Ehdr *ehdr;
- struct page *page;
- void *page_addr;
- int ret;
-
- /* only works for page backed storage */
- if (!vma->vm_file)
- return -EINVAL;
-
- page = find_get_page(vma->vm_file->f_mapping, 0);
- if (!page)
- return -EFAULT; /* page not mapped */
-
- ret = -EINVAL;
- page_addr = page_address(page);
- ehdr = (Elf32_Ehdr *)page_addr;
-
- /* compare magic x7f "ELF" */
- if (memcmp(ehdr->e_ident, ELFMAG, SELFMAG) != 0)
- goto out;
-
- /* only support executable file and shared object file */
- if (ehdr->e_type != ET_EXEC && ehdr->e_type != ET_DYN)
- goto out;
-
- if (ehdr->e_ident[EI_CLASS] == ELFCLASS32)
- ret = stack_map_get_build_id_32(page_addr, build_id);
- else if (ehdr->e_ident[EI_CLASS] == ELFCLASS64)
- ret = stack_map_get_build_id_64(page_addr, build_id);
-out:
- put_page(page);
- return ret;
-}
-
-static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
- u64 *ips, u32 trace_nr, bool user)
-{
- int i;
- struct vm_area_struct *vma;
- bool irq_work_busy = false;
- struct stack_map_irq_work *work = NULL;
-
- if (in_nmi()) {
- work = this_cpu_ptr(&up_read_work);
- if (work->irq_work.flags & IRQ_WORK_BUSY)
- /* cannot queue more up_read, fallback */
- irq_work_busy = true;
- }
-
- /*
- * We cannot do up_read() in nmi context. To do build_id lookup
- * in nmi context, we need to run up_read() in irq_work. We use
- * a percpu variable to do the irq_work. If the irq_work is
- * already used by another lookup, we fall back to report ips.
- *
- * Same fallback is used for kernel stack (!user) on a stackmap
- * with build_id.
- */
- if (!user || !current || !current->mm || irq_work_busy ||
- down_read_trylock(¤t->mm->mmap_sem) == 0) {
- /* cannot access current->mm, fall back to ips */
- for (i = 0; i < trace_nr; i++) {
- id_offs[i].status = BPF_STACK_BUILD_ID_IP;
- id_offs[i].ip = ips[i];
- }
- return;
- }
-
- for (i = 0; i < trace_nr; i++) {
- vma = find_vma(current->mm, ips[i]);
- if (!vma || stack_map_get_build_id(vma, id_offs[i].build_id)) {
- /* per entry fall back to ips */
- id_offs[i].status = BPF_STACK_BUILD_ID_IP;
- id_offs[i].ip = ips[i];
- continue;
- }
- id_offs[i].offset = (vma->vm_pgoff << PAGE_SHIFT) + ips[i]
- - vma->vm_start;
- id_offs[i].status = BPF_STACK_BUILD_ID_VALID;
- }
-
- if (!work) {
- up_read(¤t->mm->mmap_sem);
- } else {
- work->sem = ¤t->mm->mmap_sem;
- irq_work_queue(&work->irq_work);
- }
-}
-
-BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map,
- u64, flags)
-{
- struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
- struct perf_callchain_entry *trace;
- struct stack_map_bucket *bucket, *new_bucket, *old_bucket;
- u32 max_depth = map->value_size / stack_map_data_size(map);
- /* stack_map_alloc() checks that max_depth <= sysctl_perf_event_max_stack */
- u32 init_nr = sysctl_perf_event_max_stack - max_depth;
- u32 skip = flags & BPF_F_SKIP_FIELD_MASK;
- u32 hash, id, trace_nr, trace_len;
- bool user = flags & BPF_F_USER_STACK;
- bool kernel = !user;
- u64 *ips;
- bool hash_matches;
-
- if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK |
- BPF_F_FAST_STACK_CMP | BPF_F_REUSE_STACKID)))
- return -EINVAL;
-
- trace = get_perf_callchain(regs, init_nr, kernel, user,
- sysctl_perf_event_max_stack, false, false);
-
- if (unlikely(!trace))
- /* couldn't fetch the stack trace */
- return -EFAULT;
-
- /* get_perf_callchain() guarantees that trace->nr >= init_nr
- * and trace-nr <= sysctl_perf_event_max_stack, so trace_nr <= max_depth
- */
- trace_nr = trace->nr - init_nr;
-
- if (trace_nr <= skip)
- /* skipping more than usable stack trace */
- return -EFAULT;
-
- trace_nr -= skip;
- trace_len = trace_nr * sizeof(u64);
- ips = trace->ip + skip + init_nr;
- hash = jhash2((u32 *)ips, trace_len / sizeof(u32), 0);
- id = hash & (smap->n_buckets - 1);
- bucket = READ_ONCE(smap->buckets[id]);
-
- hash_matches = bucket && bucket->hash == hash;
- /* fast cmp */
- if (hash_matches && flags & BPF_F_FAST_STACK_CMP)
- return id;
-
- if (stack_map_use_build_id(map)) {
- /* for build_id+offset, pop a bucket before slow cmp */
- new_bucket = (struct stack_map_bucket *)
- pcpu_freelist_pop(&smap->freelist);
- if (unlikely(!new_bucket))
- return -ENOMEM;
- new_bucket->nr = trace_nr;
- stack_map_get_build_id_offset(
- (struct bpf_stack_build_id *)new_bucket->data,
- ips, trace_nr, user);
- trace_len = trace_nr * sizeof(struct bpf_stack_build_id);
- if (hash_matches && bucket->nr == trace_nr &&
- memcmp(bucket->data, new_bucket->data, trace_len) == 0) {
- pcpu_freelist_push(&smap->freelist, &new_bucket->fnode);
- return id;
- }
- if (bucket && !(flags & BPF_F_REUSE_STACKID)) {
- pcpu_freelist_push(&smap->freelist, &new_bucket->fnode);
- return -EEXIST;
- }
- } else {
- if (hash_matches && bucket->nr == trace_nr &&
- memcmp(bucket->data, ips, trace_len) == 0)
- return id;
- if (bucket && !(flags & BPF_F_REUSE_STACKID))
- return -EEXIST;
-
- new_bucket = (struct stack_map_bucket *)
- pcpu_freelist_pop(&smap->freelist);
- if (unlikely(!new_bucket))
- return -ENOMEM;
- memcpy(new_bucket->data, ips, trace_len);
- }
-
- new_bucket->hash = hash;
- new_bucket->nr = trace_nr;
-
- old_bucket = xchg(&smap->buckets[id], new_bucket);
- if (old_bucket)
- pcpu_freelist_push(&smap->freelist, &old_bucket->fnode);
- return id;
-}
-
-const struct bpf_func_proto bpf_get_stackid_proto = {
- .func = bpf_get_stackid,
- .gpl_only = true,
- .ret_type = RET_INTEGER,
- .arg1_type = ARG_PTR_TO_CTX,
- .arg2_type = ARG_CONST_MAP_PTR,
- .arg3_type = ARG_ANYTHING,
-};
-
-BPF_CALL_4(bpf_get_stack, struct pt_regs *, regs, void *, buf, u32, size,
- u64, flags)
-{
- u32 init_nr, trace_nr, copy_len, elem_size, num_elem;
- bool user_build_id = flags & BPF_F_USER_BUILD_ID;
- u32 skip = flags & BPF_F_SKIP_FIELD_MASK;
- bool user = flags & BPF_F_USER_STACK;
- struct perf_callchain_entry *trace;
- bool kernel = !user;
- int err = -EINVAL;
- u64 *ips;
-
- if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK |
- BPF_F_USER_BUILD_ID)))
- goto clear;
- if (kernel && user_build_id)
- goto clear;
-
- elem_size = (user && user_build_id) ? sizeof(struct bpf_stack_build_id)
- : sizeof(u64);
- if (unlikely(size % elem_size))
- goto clear;
-
- num_elem = size / elem_size;
- if (sysctl_perf_event_max_stack < num_elem)
- init_nr = 0;
- else
- init_nr = sysctl_perf_event_max_stack - num_elem;
- trace = get_perf_callchain(regs, init_nr, kernel, user,
- sysctl_perf_event_max_stack, false, false);
- if (unlikely(!trace))
- goto err_fault;
-
- trace_nr = trace->nr - init_nr;
- if (trace_nr < skip)
- goto err_fault;
-
- trace_nr -= skip;
- trace_nr = (trace_nr <= num_elem) ? trace_nr : num_elem;
- copy_len = trace_nr * elem_size;
- ips = trace->ip + skip + init_nr;
- if (user && user_build_id)
- stack_map_get_build_id_offset(buf, ips, trace_nr, user);
- else
- memcpy(buf, ips, copy_len);
-
- if (size > copy_len)
- memset(buf + copy_len, 0, size - copy_len);
- return copy_len;
-
-err_fault:
- err = -EFAULT;
-clear:
- memset(buf, 0, size);
- return err;
-}
-
-const struct bpf_func_proto bpf_get_stack_proto = {
- .func = bpf_get_stack,
- .gpl_only = true,
- .ret_type = RET_INTEGER,
- .arg1_type = ARG_PTR_TO_CTX,
- .arg2_type = ARG_PTR_TO_UNINIT_MEM,
- .arg3_type = ARG_CONST_SIZE_OR_ZERO,
- .arg4_type = ARG_ANYTHING,
-};
-
-/* Called from eBPF program */
-static void *stack_map_lookup_elem(struct bpf_map *map, void *key)
-{
- return NULL;
-}
-
-/* Called from syscall */
-int bpf_stackmap_copy(struct bpf_map *map, void *key, void *value)
-{
- struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
- struct stack_map_bucket *bucket, *old_bucket;
- u32 id = *(u32 *)key, trace_len;
-
- if (unlikely(id >= smap->n_buckets))
- return -ENOENT;
-
- bucket = xchg(&smap->buckets[id], NULL);
- if (!bucket)
- return -ENOENT;
-
- trace_len = bucket->nr * stack_map_data_size(map);
- memcpy(value, bucket->data, trace_len);
- memset(value + trace_len, 0, map->value_size - trace_len);
-
- old_bucket = xchg(&smap->buckets[id], bucket);
- if (old_bucket)
- pcpu_freelist_push(&smap->freelist, &old_bucket->fnode);
- return 0;
-}
-
-static int stack_map_get_next_key(struct bpf_map *map, void *key,
- void *next_key)
-{
- struct bpf_stack_map *smap = container_of(map,
- struct bpf_stack_map, map);
- u32 id;
-
- WARN_ON_ONCE(!rcu_read_lock_held());
-
- if (!key) {
- id = 0;
- } else {
- id = *(u32 *)key;
- if (id >= smap->n_buckets || !smap->buckets[id])
- id = 0;
- else
- id++;
- }
-
- while (id < smap->n_buckets && !smap->buckets[id])
- id++;
-
- if (id >= smap->n_buckets)
- return -ENOENT;
-
- *(u32 *)next_key = id;
- return 0;
-}
-
-static int stack_map_update_elem(struct bpf_map *map, void *key, void *value,
- u64 map_flags)
-{
- return -EINVAL;
-}
-
-/* Called from syscall or from eBPF program */
-static int stack_map_delete_elem(struct bpf_map *map, void *key)
-{
- struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
- struct stack_map_bucket *old_bucket;
- u32 id = *(u32 *)key;
-
- if (unlikely(id >= smap->n_buckets))
- return -E2BIG;
-
- old_bucket = xchg(&smap->buckets[id], NULL);
- if (old_bucket) {
- pcpu_freelist_push(&smap->freelist, &old_bucket->fnode);
- return 0;
- } else {
- return -ENOENT;
- }
-}
-
-/* Called when map->refcnt goes to zero, either from workqueue or from syscall */
-static void stack_map_free(struct bpf_map *map)
-{
- struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
-
- /* wait for bpf programs to complete before freeing stack map */
- synchronize_rcu();
-
- bpf_map_area_free(smap->elems);
- pcpu_freelist_destroy(&smap->freelist);
- bpf_map_area_free(smap);
- put_callchain_buffers();
-}
-
-const struct bpf_map_ops stack_map_ops = {
- .map_alloc = stack_map_alloc,
- .map_free = stack_map_free,
- .map_get_next_key = stack_map_get_next_key,
- .map_lookup_elem = stack_map_lookup_elem,
- .map_update_elem = stack_map_update_elem,
- .map_delete_elem = stack_map_delete_elem,
- .map_check_btf = map_check_no_btf,
-};
-
-static int __init stack_map_init(void)
-{
- int cpu;
- struct stack_map_irq_work *work;
-
- for_each_possible_cpu(cpu) {
- work = per_cpu_ptr(&up_read_work, cpu);
- init_irq_work(&work->irq_work, do_up_read);
- }
- return 0;
-}
-subsys_initcall(stack_map_init);
diff --git a/kernel/bpf/stacktracemap.c b/kernel/bpf/stacktracemap.c
new file mode 100644
index 000000000000..bb41e293418d
--- /dev/null
+++ b/kernel/bpf/stacktracemap.c
@@ -0,0 +1,624 @@
+/* Copyright (c) 2016 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#include <linux/bpf.h>
+#include <linux/jhash.h>
+#include <linux/filter.h>
+#include <linux/stacktrace.h>
+#include <linux/perf_event.h>
+#include <linux/elf.h>
+#include <linux/pagemap.h>
+#include <linux/irq_work.h>
+#include "percpu_freelist.h"
+
+#define STACK_CREATE_FLAG_MASK \
+ (BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY | \
+ BPF_F_STACK_BUILD_ID)
+
+struct stack_map_bucket {
+ struct pcpu_freelist_node fnode;
+ u32 hash;
+ u32 nr;
+ u64 data[];
+};
+
+struct bpf_stack_map {
+ struct bpf_map map;
+ void *elems;
+ struct pcpu_freelist freelist;
+ u32 n_buckets;
+ struct stack_map_bucket *buckets[];
+};
+
+/* irq_work to run up_read() for build_id lookup in nmi context */
+struct stack_map_irq_work {
+ struct irq_work irq_work;
+ struct rw_semaphore *sem;
+};
+
+static void do_up_read(struct irq_work *entry)
+{
+ struct stack_map_irq_work *work;
+
+ work = container_of(entry, struct stack_map_irq_work, irq_work);
+ up_read(work->sem);
+ work->sem = NULL;
+}
+
+static DEFINE_PER_CPU(struct stack_map_irq_work, up_read_work);
+
+static inline bool stack_map_use_build_id(struct bpf_map *map)
+{
+ return (map->map_flags & BPF_F_STACK_BUILD_ID);
+}
+
+static inline int stack_map_data_size(struct bpf_map *map)
+{
+ return stack_map_use_build_id(map) ?
+ sizeof(struct bpf_stack_build_id) : sizeof(u64);
+}
+
+static int prealloc_elems_and_freelist(struct bpf_stack_map *smap)
+{
+ u32 elem_size = sizeof(struct stack_map_bucket) + smap->map.value_size;
+ int err;
+
+ smap->elems = bpf_map_area_alloc(elem_size * smap->map.max_entries,
+ smap->map.numa_node);
+ if (!smap->elems)
+ return -ENOMEM;
+
+ err = pcpu_freelist_init(&smap->freelist);
+ if (err)
+ goto free_elems;
+
+ pcpu_freelist_populate(&smap->freelist, smap->elems, elem_size,
+ smap->map.max_entries);
+ return 0;
+
+free_elems:
+ bpf_map_area_free(smap->elems);
+ return err;
+}
+
+/* Called from syscall */
+static struct bpf_map *stack_map_alloc(union bpf_attr *attr)
+{
+ u32 value_size = attr->value_size;
+ struct bpf_stack_map *smap;
+ u64 cost, n_buckets;
+ int err;
+
+ if (!capable(CAP_SYS_ADMIN))
+ return ERR_PTR(-EPERM);
+
+ if (attr->map_flags & ~STACK_CREATE_FLAG_MASK)
+ return ERR_PTR(-EINVAL);
+
+ /* check sanity of attributes */
+ if (attr->max_entries == 0 || attr->key_size != 4 ||
+ value_size < 8 || value_size % 8)
+ return ERR_PTR(-EINVAL);
+
+ BUILD_BUG_ON(sizeof(struct bpf_stack_build_id) % sizeof(u64));
+ if (attr->map_flags & BPF_F_STACK_BUILD_ID) {
+ if (value_size % sizeof(struct bpf_stack_build_id) ||
+ value_size / sizeof(struct bpf_stack_build_id)
+ > sysctl_perf_event_max_stack)
+ return ERR_PTR(-EINVAL);
+ } else if (value_size / 8 > sysctl_perf_event_max_stack)
+ return ERR_PTR(-EINVAL);
+
+ /* hash table size must be power of 2 */
+ n_buckets = roundup_pow_of_two(attr->max_entries);
+
+ cost = n_buckets * sizeof(struct stack_map_bucket *) + sizeof(*smap);
+ if (cost >= U32_MAX - PAGE_SIZE)
+ return ERR_PTR(-E2BIG);
+
+ smap = bpf_map_area_alloc(cost, bpf_map_attr_numa_node(attr));
+ if (!smap)
+ return ERR_PTR(-ENOMEM);
+
+ err = -E2BIG;
+ cost += n_buckets * (value_size + sizeof(struct stack_map_bucket));
+ if (cost >= U32_MAX - PAGE_SIZE)
+ goto free_smap;
+
+ bpf_map_init_from_attr(&smap->map, attr);
+ smap->map.value_size = value_size;
+ smap->n_buckets = n_buckets;
+ smap->map.pages = round_up(cost, PAGE_SIZE) >> PAGE_SHIFT;
+
+ err = bpf_map_precharge_memlock(smap->map.pages);
+ if (err)
+ goto free_smap;
+
+ err = get_callchain_buffers(sysctl_perf_event_max_stack);
+ if (err)
+ goto free_smap;
+
+ err = prealloc_elems_and_freelist(smap);
+ if (err)
+ goto put_buffers;
+
+ return &smap->map;
+
+put_buffers:
+ put_callchain_buffers();
+free_smap:
+ bpf_map_area_free(smap);
+ return ERR_PTR(err);
+}
+
+#define BPF_BUILD_ID 3
+/*
+ * Parse build id from the note segment. This logic can be shared between
+ * 32-bit and 64-bit system, because Elf32_Nhdr and Elf64_Nhdr are
+ * identical.
+ */
+static inline int stack_map_parse_build_id(void *page_addr,
+ unsigned char *build_id,
+ void *note_start,
+ Elf32_Word note_size)
+{
+ Elf32_Word note_offs = 0, new_offs;
+
+ /* check for overflow */
+ if (note_start < page_addr || note_start + note_size < note_start)
+ return -EINVAL;
+
+ /* only supports note that fits in the first page */
+ if (note_start + note_size > page_addr + PAGE_SIZE)
+ return -EINVAL;
+
+ while (note_offs + sizeof(Elf32_Nhdr) < note_size) {
+ Elf32_Nhdr *nhdr = (Elf32_Nhdr *)(note_start + note_offs);
+
+ if (nhdr->n_type == BPF_BUILD_ID &&
+ nhdr->n_namesz == sizeof("GNU") &&
+ nhdr->n_descsz == BPF_BUILD_ID_SIZE) {
+ memcpy(build_id,
+ note_start + note_offs +
+ ALIGN(sizeof("GNU"), 4) + sizeof(Elf32_Nhdr),
+ BPF_BUILD_ID_SIZE);
+ return 0;
+ }
+ new_offs = note_offs + sizeof(Elf32_Nhdr) +
+ ALIGN(nhdr->n_namesz, 4) + ALIGN(nhdr->n_descsz, 4);
+ if (new_offs <= note_offs) /* overflow */
+ break;
+ note_offs = new_offs;
+ }
+ return -EINVAL;
+}
+
+/* Parse build ID from 32-bit ELF */
+static int stack_map_get_build_id_32(void *page_addr,
+ unsigned char *build_id)
+{
+ Elf32_Ehdr *ehdr = (Elf32_Ehdr *)page_addr;
+ Elf32_Phdr *phdr;
+ int i;
+
+ /* only supports phdr that fits in one page */
+ if (ehdr->e_phnum >
+ (PAGE_SIZE - sizeof(Elf32_Ehdr)) / sizeof(Elf32_Phdr))
+ return -EINVAL;
+
+ phdr = (Elf32_Phdr *)(page_addr + sizeof(Elf32_Ehdr));
+
+ for (i = 0; i < ehdr->e_phnum; ++i)
+ if (phdr[i].p_type == PT_NOTE)
+ return stack_map_parse_build_id(page_addr, build_id,
+ page_addr + phdr[i].p_offset,
+ phdr[i].p_filesz);
+ return -EINVAL;
+}
+
+/* Parse build ID from 64-bit ELF */
+static int stack_map_get_build_id_64(void *page_addr,
+ unsigned char *build_id)
+{
+ Elf64_Ehdr *ehdr = (Elf64_Ehdr *)page_addr;
+ Elf64_Phdr *phdr;
+ int i;
+
+ /* only supports phdr that fits in one page */
+ if (ehdr->e_phnum >
+ (PAGE_SIZE - sizeof(Elf64_Ehdr)) / sizeof(Elf64_Phdr))
+ return -EINVAL;
+
+ phdr = (Elf64_Phdr *)(page_addr + sizeof(Elf64_Ehdr));
+
+ for (i = 0; i < ehdr->e_phnum; ++i)
+ if (phdr[i].p_type == PT_NOTE)
+ return stack_map_parse_build_id(page_addr, build_id,
+ page_addr + phdr[i].p_offset,
+ phdr[i].p_filesz);
+ return -EINVAL;
+}
+
+/* Parse build ID of ELF file mapped to vma */
+static int stack_map_get_build_id(struct vm_area_struct *vma,
+ unsigned char *build_id)
+{
+ Elf32_Ehdr *ehdr;
+ struct page *page;
+ void *page_addr;
+ int ret;
+
+ /* only works for page backed storage */
+ if (!vma->vm_file)
+ return -EINVAL;
+
+ page = find_get_page(vma->vm_file->f_mapping, 0);
+ if (!page)
+ return -EFAULT; /* page not mapped */
+
+ ret = -EINVAL;
+ page_addr = page_address(page);
+ ehdr = (Elf32_Ehdr *)page_addr;
+
+ /* compare magic x7f "ELF" */
+ if (memcmp(ehdr->e_ident, ELFMAG, SELFMAG) != 0)
+ goto out;
+
+ /* only support executable file and shared object file */
+ if (ehdr->e_type != ET_EXEC && ehdr->e_type != ET_DYN)
+ goto out;
+
+ if (ehdr->e_ident[EI_CLASS] == ELFCLASS32)
+ ret = stack_map_get_build_id_32(page_addr, build_id);
+ else if (ehdr->e_ident[EI_CLASS] == ELFCLASS64)
+ ret = stack_map_get_build_id_64(page_addr, build_id);
+out:
+ put_page(page);
+ return ret;
+}
+
+static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
+ u64 *ips, u32 trace_nr, bool user)
+{
+ int i;
+ struct vm_area_struct *vma;
+ bool irq_work_busy = false;
+ struct stack_map_irq_work *work = NULL;
+
+ if (in_nmi()) {
+ work = this_cpu_ptr(&up_read_work);
+ if (work->irq_work.flags & IRQ_WORK_BUSY)
+ /* cannot queue more up_read, fallback */
+ irq_work_busy = true;
+ }
+
+ /*
+ * We cannot do up_read() in nmi context. To do build_id lookup
+ * in nmi context, we need to run up_read() in irq_work. We use
+ * a percpu variable to do the irq_work. If the irq_work is
+ * already used by another lookup, we fall back to report ips.
+ *
+ * Same fallback is used for kernel stack (!user) on a stackmap
+ * with build_id.
+ */
+ if (!user || !current || !current->mm || irq_work_busy ||
+ down_read_trylock(¤t->mm->mmap_sem) == 0) {
+ /* cannot access current->mm, fall back to ips */
+ for (i = 0; i < trace_nr; i++) {
+ id_offs[i].status = BPF_STACK_BUILD_ID_IP;
+ id_offs[i].ip = ips[i];
+ }
+ return;
+ }
+
+ for (i = 0; i < trace_nr; i++) {
+ vma = find_vma(current->mm, ips[i]);
+ if (!vma || stack_map_get_build_id(vma, id_offs[i].build_id)) {
+ /* per entry fall back to ips */
+ id_offs[i].status = BPF_STACK_BUILD_ID_IP;
+ id_offs[i].ip = ips[i];
+ continue;
+ }
+ id_offs[i].offset = (vma->vm_pgoff << PAGE_SHIFT) + ips[i]
+ - vma->vm_start;
+ id_offs[i].status = BPF_STACK_BUILD_ID_VALID;
+ }
+
+ if (!work) {
+ up_read(¤t->mm->mmap_sem);
+ } else {
+ work->sem = ¤t->mm->mmap_sem;
+ irq_work_queue(&work->irq_work);
+ }
+}
+
+BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map,
+ u64, flags)
+{
+ struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
+ struct perf_callchain_entry *trace;
+ struct stack_map_bucket *bucket, *new_bucket, *old_bucket;
+ u32 max_depth = map->value_size / stack_map_data_size(map);
+ /* stack_map_alloc() checks that max_depth <= sysctl_perf_event_max_stack */
+ u32 init_nr = sysctl_perf_event_max_stack - max_depth;
+ u32 skip = flags & BPF_F_SKIP_FIELD_MASK;
+ u32 hash, id, trace_nr, trace_len;
+ bool user = flags & BPF_F_USER_STACK;
+ bool kernel = !user;
+ u64 *ips;
+ bool hash_matches;
+
+ if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK |
+ BPF_F_FAST_STACK_CMP | BPF_F_REUSE_STACKID)))
+ return -EINVAL;
+
+ trace = get_perf_callchain(regs, init_nr, kernel, user,
+ sysctl_perf_event_max_stack, false, false);
+
+ if (unlikely(!trace))
+ /* couldn't fetch the stack trace */
+ return -EFAULT;
+
+ /* get_perf_callchain() guarantees that trace->nr >= init_nr
+ * and trace-nr <= sysctl_perf_event_max_stack, so trace_nr <= max_depth
+ */
+ trace_nr = trace->nr - init_nr;
+
+ if (trace_nr <= skip)
+ /* skipping more than usable stack trace */
+ return -EFAULT;
+
+ trace_nr -= skip;
+ trace_len = trace_nr * sizeof(u64);
+ ips = trace->ip + skip + init_nr;
+ hash = jhash2((u32 *)ips, trace_len / sizeof(u32), 0);
+ id = hash & (smap->n_buckets - 1);
+ bucket = READ_ONCE(smap->buckets[id]);
+
+ hash_matches = bucket && bucket->hash == hash;
+ /* fast cmp */
+ if (hash_matches && flags & BPF_F_FAST_STACK_CMP)
+ return id;
+
+ if (stack_map_use_build_id(map)) {
+ /* for build_id+offset, pop a bucket before slow cmp */
+ new_bucket = (struct stack_map_bucket *)
+ pcpu_freelist_pop(&smap->freelist);
+ if (unlikely(!new_bucket))
+ return -ENOMEM;
+ new_bucket->nr = trace_nr;
+ stack_map_get_build_id_offset(
+ (struct bpf_stack_build_id *)new_bucket->data,
+ ips, trace_nr, user);
+ trace_len = trace_nr * sizeof(struct bpf_stack_build_id);
+ if (hash_matches && bucket->nr == trace_nr &&
+ memcmp(bucket->data, new_bucket->data, trace_len) == 0) {
+ pcpu_freelist_push(&smap->freelist, &new_bucket->fnode);
+ return id;
+ }
+ if (bucket && !(flags & BPF_F_REUSE_STACKID)) {
+ pcpu_freelist_push(&smap->freelist, &new_bucket->fnode);
+ return -EEXIST;
+ }
+ } else {
+ if (hash_matches && bucket->nr == trace_nr &&
+ memcmp(bucket->data, ips, trace_len) == 0)
+ return id;
+ if (bucket && !(flags & BPF_F_REUSE_STACKID))
+ return -EEXIST;
+
+ new_bucket = (struct stack_map_bucket *)
+ pcpu_freelist_pop(&smap->freelist);
+ if (unlikely(!new_bucket))
+ return -ENOMEM;
+ memcpy(new_bucket->data, ips, trace_len);
+ }
+
+ new_bucket->hash = hash;
+ new_bucket->nr = trace_nr;
+
+ old_bucket = xchg(&smap->buckets[id], new_bucket);
+ if (old_bucket)
+ pcpu_freelist_push(&smap->freelist, &old_bucket->fnode);
+ return id;
+}
+
+const struct bpf_func_proto bpf_get_stackid_proto = {
+ .func = bpf_get_stackid,
+ .gpl_only = true,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_PTR_TO_CTX,
+ .arg2_type = ARG_CONST_MAP_PTR,
+ .arg3_type = ARG_ANYTHING,
+};
+
+BPF_CALL_4(bpf_get_stack, struct pt_regs *, regs, void *, buf, u32, size,
+ u64, flags)
+{
+ u32 init_nr, trace_nr, copy_len, elem_size, num_elem;
+ bool user_build_id = flags & BPF_F_USER_BUILD_ID;
+ u32 skip = flags & BPF_F_SKIP_FIELD_MASK;
+ bool user = flags & BPF_F_USER_STACK;
+ struct perf_callchain_entry *trace;
+ bool kernel = !user;
+ int err = -EINVAL;
+ u64 *ips;
+
+ if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK |
+ BPF_F_USER_BUILD_ID)))
+ goto clear;
+ if (kernel && user_build_id)
+ goto clear;
+
+ elem_size = (user && user_build_id) ? sizeof(struct bpf_stack_build_id)
+ : sizeof(u64);
+ if (unlikely(size % elem_size))
+ goto clear;
+
+ num_elem = size / elem_size;
+ if (sysctl_perf_event_max_stack < num_elem)
+ init_nr = 0;
+ else
+ init_nr = sysctl_perf_event_max_stack - num_elem;
+ trace = get_perf_callchain(regs, init_nr, kernel, user,
+ sysctl_perf_event_max_stack, false, false);
+ if (unlikely(!trace))
+ goto err_fault;
+
+ trace_nr = trace->nr - init_nr;
+ if (trace_nr < skip)
+ goto err_fault;
+
+ trace_nr -= skip;
+ trace_nr = (trace_nr <= num_elem) ? trace_nr : num_elem;
+ copy_len = trace_nr * elem_size;
+ ips = trace->ip + skip + init_nr;
+ if (user && user_build_id)
+ stack_map_get_build_id_offset(buf, ips, trace_nr, user);
+ else
+ memcpy(buf, ips, copy_len);
+
+ if (size > copy_len)
+ memset(buf + copy_len, 0, size - copy_len);
+ return copy_len;
+
+err_fault:
+ err = -EFAULT;
+clear:
+ memset(buf, 0, size);
+ return err;
+}
+
+const struct bpf_func_proto bpf_get_stack_proto = {
+ .func = bpf_get_stack,
+ .gpl_only = true,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_PTR_TO_CTX,
+ .arg2_type = ARG_PTR_TO_UNINIT_MEM,
+ .arg3_type = ARG_CONST_SIZE_OR_ZERO,
+ .arg4_type = ARG_ANYTHING,
+};
+
+/* Called from eBPF program */
+static void *stack_map_lookup_elem(struct bpf_map *map, void *key)
+{
+ return NULL;
+}
+
+/* Called from syscall */
+int bpf_stackmap_copy(struct bpf_map *map, void *key, void *value)
+{
+ struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
+ struct stack_map_bucket *bucket, *old_bucket;
+ u32 id = *(u32 *)key, trace_len;
+
+ if (unlikely(id >= smap->n_buckets))
+ return -ENOENT;
+
+ bucket = xchg(&smap->buckets[id], NULL);
+ if (!bucket)
+ return -ENOENT;
+
+ trace_len = bucket->nr * stack_map_data_size(map);
+ memcpy(value, bucket->data, trace_len);
+ memset(value + trace_len, 0, map->value_size - trace_len);
+
+ old_bucket = xchg(&smap->buckets[id], bucket);
+ if (old_bucket)
+ pcpu_freelist_push(&smap->freelist, &old_bucket->fnode);
+ return 0;
+}
+
+static int stack_map_get_next_key(struct bpf_map *map, void *key,
+ void *next_key)
+{
+ struct bpf_stack_map *smap = container_of(map,
+ struct bpf_stack_map, map);
+ u32 id;
+
+ WARN_ON_ONCE(!rcu_read_lock_held());
+
+ if (!key) {
+ id = 0;
+ } else {
+ id = *(u32 *)key;
+ if (id >= smap->n_buckets || !smap->buckets[id])
+ id = 0;
+ else
+ id++;
+ }
+
+ while (id < smap->n_buckets && !smap->buckets[id])
+ id++;
+
+ if (id >= smap->n_buckets)
+ return -ENOENT;
+
+ *(u32 *)next_key = id;
+ return 0;
+}
+
+static int stack_map_update_elem(struct bpf_map *map, void *key, void *value,
+ u64 map_flags)
+{
+ return -EINVAL;
+}
+
+/* Called from syscall or from eBPF program */
+static int stack_map_delete_elem(struct bpf_map *map, void *key)
+{
+ struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
+ struct stack_map_bucket *old_bucket;
+ u32 id = *(u32 *)key;
+
+ if (unlikely(id >= smap->n_buckets))
+ return -E2BIG;
+
+ old_bucket = xchg(&smap->buckets[id], NULL);
+ if (old_bucket) {
+ pcpu_freelist_push(&smap->freelist, &old_bucket->fnode);
+ return 0;
+ } else {
+ return -ENOENT;
+ }
+}
+
+/* Called when map->refcnt goes to zero, either from workqueue or from syscall */
+static void stack_map_free(struct bpf_map *map)
+{
+ struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
+
+ /* wait for bpf programs to complete before freeing stack map */
+ synchronize_rcu();
+
+ bpf_map_area_free(smap->elems);
+ pcpu_freelist_destroy(&smap->freelist);
+ bpf_map_area_free(smap);
+ put_callchain_buffers();
+}
+
+const struct bpf_map_ops stack_trace_map_ops = {
+ .map_alloc = stack_map_alloc,
+ .map_free = stack_map_free,
+ .map_get_next_key = stack_map_get_next_key,
+ .map_lookup_elem = stack_map_lookup_elem,
+ .map_update_elem = stack_map_update_elem,
+ .map_delete_elem = stack_map_delete_elem,
+ .map_check_btf = map_check_no_btf,
+};
+
+static int __init stack_map_init(void)
+{
+ int cpu;
+ struct stack_map_irq_work *work;
+
+ for_each_possible_cpu(cpu) {
+ work = per_cpu_ptr(&up_read_work, cpu);
+ init_irq_work(&work->irq_work, do_up_read);
+ }
+ return 0;
+}
+subsys_initcall(stack_map_init);
^ permalink raw reply related
* [RFC PATCH bpf-next v3 2/7] bpf/syscall: allow key to be null in map functions
From: Mauricio Vasquez B @ 2018-09-18 4:52 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, netdev; +Cc: Yonghong Song
In-Reply-To: <153724634652.7866.6354309647800281793.stgit@kernel>
This commit adds the required logic to allow key being NULL
in case the key_size of the map is 0.
A new __bpf_copy_key function helper only copies the key from
userpsace when key_size != 0, otherwise it enforces that key must be
null.
Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>
---
kernel/bpf/syscall.c | 19 +++++++++++++++----
1 file changed, 15 insertions(+), 4 deletions(-)
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 3c9636f03bb2..f2d4e4f280dc 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -651,6 +651,17 @@ int __weak bpf_stackmap_copy(struct bpf_map *map, void *key, void *value)
return -ENOTSUPP;
}
+static void *__bpf_copy_key(void __user *ukey, u64 key_size)
+{
+ if (key_size)
+ return memdup_user(ukey, key_size);
+
+ if (ukey)
+ return ERR_PTR(-EINVAL);
+
+ return NULL;
+}
+
/* last field in 'union bpf_attr' used by this command */
#define BPF_MAP_LOOKUP_ELEM_LAST_FIELD value
@@ -678,7 +689,7 @@ static int map_lookup_elem(union bpf_attr *attr)
goto err_put;
}
- key = memdup_user(ukey, map->key_size);
+ key = __bpf_copy_key(ukey, map->key_size);
if (IS_ERR(key)) {
err = PTR_ERR(key);
goto err_put;
@@ -766,7 +777,7 @@ static int map_update_elem(union bpf_attr *attr)
goto err_put;
}
- key = memdup_user(ukey, map->key_size);
+ key = __bpf_copy_key(ukey, map->key_size);
if (IS_ERR(key)) {
err = PTR_ERR(key);
goto err_put;
@@ -864,7 +875,7 @@ static int map_delete_elem(union bpf_attr *attr)
goto err_put;
}
- key = memdup_user(ukey, map->key_size);
+ key = __bpf_copy_key(ukey, map->key_size);
if (IS_ERR(key)) {
err = PTR_ERR(key);
goto err_put;
@@ -916,7 +927,7 @@ static int map_get_next_key(union bpf_attr *attr)
}
if (ukey) {
- key = memdup_user(ukey, map->key_size);
+ key = __bpf_copy_key(ukey, map->key_size);
if (IS_ERR(key)) {
err = PTR_ERR(key);
goto err_put;
^ permalink raw reply related
* Re: kernel 4.18.5 Realtek 8111G network adapter stops responding under high system load
From: David Arendt @ 2018-09-18 10:23 UTC (permalink / raw)
To: Maciej S. Szmigiero; +Cc: linux-kernel, nic_swsd, netdev
In-Reply-To: <6c14f6d0-ea61-b8e6-57a2-940d32330ed2@maciej.szmigiero.name>
Hi,
Today I had the network adapter problems again. So the patch doesn't seem to change anything regarding this problem. This week my time is unfortunately very limited, but I will try to find some time next weekend to look a bit more into the issue.
Thanks in advance,
David Arendt
Maciej S. Szmigiero – Sun, 16. September 2018 2:12
> [ I've added Realtek Linux NIC and netdev mailing lists to CC ]
>
> Hi David,
>
> On 15.09.2018 23:23, David Arendt wrote:
> > Hi,
> >
> > just a follow up:
> >
> > In kernel 4.18.8 the behaviour is different.
> >
> > The network is not reachable a number of times, but restarting to be
> > reachable by itself before it finally is no longer reachable at all.
> >
> > Here the logging output:
> >
> > Sep 15 17:44:43 server kernel: NETDEV WATCHDOG: enp3s0 (r8169): transmit
> > queue 0 timed out
> > Sep 15 17:44:43 server kernel: r8169 0000:03:00.0 enp3s0: link up
> > Sep 15 18:10:26 server kernel: r8169 0000:03:00.0 enp3s0: link up
> > Sep 15 18:12:24 server kernel: r8169 0000:03:00.0 enp3s0: link up
> > Sep 15 18:13:19 server kernel: r8169 0000:03:00.0 enp3s0: link up
> > Sep 15 18:14:48 server kernel: r8169 0000:03:00.0 enp3s0: link up
> > Sep 15 18:20:24 server kernel: r8169 0000:03:00.0 enp3s0: link up
> > Sep 15 18:34:19 server kernel: r8169 0000:03:00.0 enp3s0: link up
> > Sep 15 18:43:43 server kernel: r8169 0000:03:00.0 enp3s0: link up
> > Sep 15 18:46:26 server kernel: r8169 0000:03:00.0 enp3s0: link up
> > Sep 15 19:00:24 server kernel: r8169 0000:03:00.0 enp3s0: link up
> >
> > From 17:44 ro 18:46 the network is recovering automatically. After the
> > up from 19:00, the network is no longer reachable without any additional
> > message.
> >
> > If looking at ifconfig, the counter for TX packets is incrementing, the
> > counter for RX packets not.
> >
> > Here again the driver from 4.17.14 is working flawlessly.
>
> Could you please try this patch on top of 4.18.8:
> git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/
>
> In my case the problem fixed by the above commit was only limited to
> bad TX performance but my r8169 NIC models were different from what
> you have.
>
> If this does not help then try bisecting the issue
> (maybe limited to drivers/net/ethernet/realtek/r8169.c to save time).
> If the NIC dies after a heavy load it might be possible to generate
> such load quickly by in-kernel pktgen.
>
> If that's not possible then at please least compare NIC register
> values displayed by "ethtool -d enp3s0" between working and
> non-working kernels.
>
> > Thanks in advance,
> > David Arendt
>
> Maciej
>
> >
> >
> > On 9/4/18 8:19 AM, David Arendt wrote:
> >> Hi,
> >>
> >> When using kernel 4.18.5 the Realtek 8111G network adapter stops
> >> responding under high system load.
> >>
> >> Dmesg is showing no errors.
> >>
> >> Sometimes an ifconfig enp3s0 down followed by an ifconfig enp3s0 up is
> >> enough for the network adapter to restart responding. Sometimes a reboot
> >> is necessary.
> >>
> >> When copying r8169.c from 4.17.14 to the 4.18.5 kernel, networking works
> >> perfectly stable on 4.18.5 so the problem seems r8169.c related.
> >>
> >> Here the output from lshw:
> >>
> >> *-pci:2
> >> description: PCI bridge
> >> product: 8 Series/C220 Series Chipset Family PCI Express
> >> Root Port #3
> >> vendor: Intel Corporation
> >> physical id: 1c.2
> >> bus info: pci@0000:00:1c.2
> >> version: d5
> >> width: 32 bits
> >> clock: 33MHz
> >> capabilities: pci pciexpress msi pm normal_decode
> >> bus_master cap_list
> >> configuration: driver=pcieport
> >> resources: irq:18 ioport:d000(size=4096)
> >> memory:f7300000-f73fffff ioport:f2100000(size=1048576)
> >> *-network
> >> description: Ethernet interface
> >> product: RTL8111/8168/8411 PCI Express Gigabit Ethernet
> >> Controller
> >> vendor: Realtek Semiconductor Co., Ltd.
> >> physical id: 0
> >> bus info: pci@0000:03:00.0
> >> logical name: enp3s0
> >> version: 0c
> >> serial: <hidden>
> >> size: 1Gbit/s
> >> capacity: 1Gbit/s
> >> width: 64 bits
> >> clock: 33MHz
> >> capabilities: pm msi pciexpress msix vpd bus_master
> >> cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt
> >> 1000bt-fd autonegotiation
> >> configuration: autonegotiation=on broadcast=yes
> >> driver=r8169 driverversion=2.3LK-NAPI duplex=full
> >> firmware=rtl8168g-2_0.0.1 02/06/13 latency=0 link=yes multicast=yes
> >> port=MII speed=1Gbit/s
> >> resources: irq:18 ioport:d000(size=256)
> >> memory:f7300000-f7300fff memory:f2100000-f2103fff
> >>
> >> Thanks in advance for looking into this,
> >>
> >> David Arendt
> >>
> >>
> >
^ permalink raw reply
* [RFC PATCH bpf-next v3 0/7] Implement bpf queue/stack maps
From: Mauricio Vasquez B @ 2018-09-18 4:52 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, netdev; +Cc: Yonghong Song
In some applications this is needed have a pool of free elements, like for
example the list of free L4 ports in a SNAT. None of the current maps allow
to do it as it is not possibleto get an any element without having they key
it is associated to.
This patchset implements two new kind of eBPF maps: queue and stack.
Those maps provide to eBPF programs the peek, push and pop operations, and for
userspace applications a new bpf_map_lookup_and_delete_elem() is added.
Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>
v2 -> v3:
- Return elements by value instead that by reference
- Implement queue/stack base on array and head + tail indexes
- Rename stack trace related files to avoid confusion and conflicts
v1 -> v2:
- Create two separate maps instead of single one + flags
- Implement bpf_map_lookup_and_delete syscall
- Support peek operation
- Define replacement policy through flags in the update() method
- Add eBPF side tests
---
Mauricio Vasquez B (7):
bpf: rename stack trace map
bpf/syscall: allow key to be null in map functions
bpf: add lookup_and_delete map operation
bpf: add bpf queue and stack maps
bpf: restrict use of peek/push/pop
Sync uapi/bpf.h to tools/include
selftests/bpf: add test cases for queue and stack maps
include/linux/bpf.h | 4
include/linux/bpf_types.h | 4
include/uapi/linux/bpf.h | 31 +
kernel/bpf/Makefile | 4
kernel/bpf/core.c | 3
kernel/bpf/helpers.c | 98 +++
kernel/bpf/queue_stack_maps.c | 291 +++++++++
kernel/bpf/stackmap.c | 624 --------------------
kernel/bpf/stacktracemap.c | 624 ++++++++++++++++++++
kernel/bpf/syscall.c | 101 +++
kernel/bpf/verifier.c | 19 +
net/core/filter.c | 6
tools/include/uapi/linux/bpf.h | 31 +
tools/lib/bpf/bpf.c | 12
tools/lib/bpf/bpf.h | 1
tools/testing/selftests/bpf/Makefile | 5
tools/testing/selftests/bpf/bpf_helpers.h | 7
tools/testing/selftests/bpf/test_maps.c | 130 ++++
tools/testing/selftests/bpf/test_progs.c | 99 +++
tools/testing/selftests/bpf/test_queue_map.c | 4
tools/testing/selftests/bpf/test_queue_stack_map.h | 59 ++
tools/testing/selftests/bpf/test_stack_map.c | 4
22 files changed, 1526 insertions(+), 635 deletions(-)
create mode 100644 kernel/bpf/queue_stack_maps.c
delete mode 100644 kernel/bpf/stackmap.c
create mode 100644 kernel/bpf/stacktracemap.c
create mode 100644 tools/testing/selftests/bpf/test_queue_map.c
create mode 100644 tools/testing/selftests/bpf/test_queue_stack_map.h
create mode 100644 tools/testing/selftests/bpf/test_stack_map.c
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox