* Re: OFT - reserving CPU's for networking
From: Martin Josefsson @ 2010-05-01 20:31 UTC (permalink / raw)
To: David Miller; +Cc: tglx, shemminger, eric.dumazet, ak, netdev, andi, peterz
In-Reply-To: <20100430.115715.216750975.davem@davemloft.net>
On Fri, 30 Apr 2010, David Miller wrote:
> Then we can do cool tricks like having the cpu spin on a mwait() on the
> network device's status descriptor in memory.
Can you have mwait monitor multiple cachelines for stores? If not then it
might be hard to do that when you have multiple nics and you actually
need to use the status descriptors, otherwise you could possibly have them
all written to the same cacheline.
Or if the nic doesn't support updating a status descriptor in memory.
If you just want to wake up quickly without using interrupts it might be
possible to abuse MSI to wake up without actually using interrupts, set
the address to the cacheline that is being monitored.
/Martin
^ permalink raw reply
* Re: [PATCH net-next-2.6] net: sock_def_readable() and friends RCU conversion
From: David Miller @ 2010-05-01 22:00 UTC (permalink / raw)
To: eric.dumazet; +Cc: hadi, xiaosuo, therbert, shemminger, netdev, eilong, bmb
In-Reply-To: <1272701011.2230.134.camel@edumazet-laptop>
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Sat, 01 May 2010 10:03:31 +0200
> David, I also need this RCU thing in order to be able to group all
> wakeups at the end of net_rx_action().
>
> Plan was to use RCU, so that I dont need to increase sk_refcnt when
> queueing a "wakeup" (and decrease sk_refcnt a long time after)
>
> Previous attempt was a bit hacky,
> http://patchwork.ozlabs.org/patch/24179/
>
> I expect 2010 one will be cleaner :)
Fair enough, I'm convinced now, applied thanks!
^ permalink raw reply
* Re: OFT - reserving CPU's for networking
From: David Miller @ 2010-05-01 22:03 UTC (permalink / raw)
To: andi; +Cc: tglx, shemminger, eric.dumazet, netdev, peterz
In-Reply-To: <20100501105304.GA9434@gargoyle.fritz.box>
From: Andi Kleen <andi@firstfloor.org>
Date: Sat, 1 May 2010 12:53:04 +0200
>> And we don't want it to, because the decision mechanisms for steering
>> that we using now are starting to get into the stateful territory and
>> that's verbotton for NIC offload as far as we're concerned.
>
> Huh? I thought full TCP offload was forbidden?[1] Statefull as in NIC
> (or someone else like netfilter) tracking flows is quite common and very far
> from full offload. AFAIK it doesn't have near all the problems full
> offload has.
We're tracking flow cpu location state at the socket operations, like
recvmsg() and sendmsg(), where it belongs.
Would you like us to call into the card drivers and firmware at these
spots instead?
^ permalink raw reply
* Re: OFT - reserving CPU's for networking
From: David Miller @ 2010-05-01 22:13 UTC (permalink / raw)
To: gandalf; +Cc: tglx, shemminger, eric.dumazet, ak, netdev, andi, peterz
In-Reply-To: <Pine.LNX.4.62.1005012222320.24624@wlug.westbo.se>
From: Martin Josefsson <gandalf@mjufs.se>
Date: Sat, 1 May 2010 22:31:05 +0200 (CEST)
> On Fri, 30 Apr 2010, David Miller wrote:
>
>> Then we can do cool tricks like having the cpu spin on a mwait() on
>> the
>> network device's status descriptor in memory.
>
> Can you have mwait monitor multiple cachelines for stores?
The idea is that if you have hundreds of cpus threads (several of my
machines do, and it's not too long before these kinds of boxes will be
common) in your machine you can spare one for each NIC.
^ permalink raw reply
* Re: OFT - reserving CPU's for networking
From: Andi Kleen @ 2010-05-01 22:58 UTC (permalink / raw)
To: David Miller; +Cc: tglx, shemminger, eric.dumazet, netdev, peterz
In-Reply-To: <20100501.150338.93457735.davem@davemloft.net>
> We're tracking flow cpu location state at the socket operations, like
> recvmsg() and sendmsg(), where it belongs.
>
> Would you like us to call into the card drivers and firmware at these
> spots instead?
No, that's not needed for lazy flow tracking like in netfilter or
some NICs, it doesn't need exact updates. It just works with seen network
packets.
-Andi
^ permalink raw reply
* Re: OFT - reserving CPU's for networking
From: David Miller @ 2010-05-01 23:29 UTC (permalink / raw)
To: andi; +Cc: tglx, shemminger, eric.dumazet, netdev, peterz
In-Reply-To: <20100501225815.GA8074@gargoyle.fritz.box>
From: Andi Kleen <andi@firstfloor.org>
Date: Sun, 2 May 2010 00:58:15 +0200
>> We're tracking flow cpu location state at the socket operations, like
>> recvmsg() and sendmsg(), where it belongs.
>>
>> Would you like us to call into the card drivers and firmware at these
>> spots instead?
>
> No, that's not needed for lazy flow tracking like in netfilter or
> some NICs, it doesn't need exact updates. It just works with seen network
> packets.
Well what we need is exact flow updates so that we steer packets
to where the applications actually are.
Andi, this discussion is going in circles, can I just say "yeah you're
right Andi" and this will satisfy your desire to be correct and we can
be done with this?
Thanks.
^ permalink raw reply
* Re: OFT - reserving CPU's for networking
From: Ben Hutchings @ 2010-05-01 23:44 UTC (permalink / raw)
To: David Miller; +Cc: andi, tglx, shemminger, eric.dumazet, netdev, peterz
In-Reply-To: <20100501.150338.93457735.davem@davemloft.net>
On Sat, 2010-05-01 at 15:03 -0700, David Miller wrote:
> From: Andi Kleen <andi@firstfloor.org>
> Date: Sat, 1 May 2010 12:53:04 +0200
>
> >> And we don't want it to, because the decision mechanisms for steering
> >> that we using now are starting to get into the stateful territory and
> >> that's verbotton for NIC offload as far as we're concerned.
> >
> > Huh? I thought full TCP offload was forbidden?[1] Statefull as in NIC
> > (or someone else like netfilter) tracking flows is quite common and very far
> > from full offload. AFAIK it doesn't have near all the problems full
> > offload has.
>
> We're tracking flow cpu location state at the socket operations, like
> recvmsg() and sendmsg(), where it belongs.
>
> Would you like us to call into the card drivers and firmware at these
> spots instead?
I'm interested in experimenting with this at some point, since our
hardware supports a fairly large number of filters that could be used
for it.
Ben.
--
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply
* Re: [PATCH 1/1] net/usb: initiate sync sequence in sierra_net.c driver
From: David Miller @ 2010-05-02 1:08 UTC (permalink / raw)
To: epasheva-ywE8TTl5eJHWpu6QEFMNjNBPR1lH4CV8
Cc: dbrownell-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f,
rfiler-ywE8TTl5eJHWpu6QEFMNjNBPR1lH4CV8,
linux-usb-u79uwXL29TY76Z2rM5mHXA, netdev-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1272497304.8835.2.camel@Linuxdev4-laptop>
From: Elina Pasheva <epasheva-ywE8TTl5eJHWpu6QEFMNjNBPR1lH4CV8@public.gmane.org>
Date: Wed, 28 Apr 2010 16:28:24 -0700
> Subject: [PATCH 1/1] net/usb: initiate sync sequence in sierra_net.c driver
> From: Elina Pasheva <epasheva-ywE8TTl5eJHWpu6QEFMNjNBPR1lH4CV8@public.gmane.org>
>
> The following patch adds the initiation of the sync sequence to
> "sierra_net_bind()". If this step is omitted, the modem will never sync up
> with the host and it will not be possible to establish a data connection.
> This is a high priority patch.
>
> This patch has been checked against net-2.6 tree.
> Signed-off-by: Elina Pasheva <epasheva-ywE8TTl5eJHWpu6QEFMNjNBPR1lH4CV8@public.gmane.org>
> Signed-off-by: Rory Filer <rfiler-ywE8TTl5eJHWpu6QEFMNjNBPR1lH4CV8@public.gmane.org>
> Tested-by: Elina Pasheva <epasheva-ywE8TTl5eJHWpu6QEFMNjNBPR1lH4CV8@public.gmane.org>
Applied.
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH net-next-2.6] net: eth_type_trans() should inline skb_pull()
From: David Miller @ 2010-05-02 1:15 UTC (permalink / raw)
To: eric.dumazet; +Cc: netdev, therbert, hadi
In-Reply-To: <1272696145.2230.101.camel@edumazet-laptop>
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Sat, 01 May 2010 08:42:25 +0200
> [PATCH net-next-2.6] net: eth_type_trans() should inline skb_pull()
>
> With RPS, this patch can give a 5 % boost in performance.
>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Awesome, but let's do this in a way that allows us to easily annotate
where inlining makes sense in other places, not just here.
Something like this, ok?
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 82f5116..746a652 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -1128,6 +1128,11 @@ static inline unsigned char *__skb_pull(struct sk_buff *skb, unsigned int len)
return skb->data += len;
}
+static inline unsigned char *skb_pull_inline(struct sk_buff *skb, unsigned int len)
+{
+ return unlikely(len > skb->len) ? NULL : __skb_pull(skb, len);
+}
+
extern unsigned char *__pskb_pull_tail(struct sk_buff *skb, int delta);
static inline unsigned char *__pskb_pull(struct sk_buff *skb, unsigned int len)
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 4218ff4..8b9c109 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -1051,7 +1051,7 @@ EXPORT_SYMBOL(skb_push);
*/
unsigned char *skb_pull(struct sk_buff *skb, unsigned int len)
{
- return unlikely(len > skb->len) ? NULL : __skb_pull(skb, len);
+ return skb_pull_inline(skb, len);
}
EXPORT_SYMBOL(skb_pull);
diff --git a/net/ethernet/eth.c b/net/ethernet/eth.c
index 0c0d272..61ec032 100644
--- a/net/ethernet/eth.c
+++ b/net/ethernet/eth.c
@@ -162,7 +162,7 @@ __be16 eth_type_trans(struct sk_buff *skb, struct net_device *dev)
skb->dev = dev;
skb_reset_mac_header(skb);
- skb_pull(skb, ETH_HLEN);
+ skb_pull_inline(skb, ETH_HLEN);
eth = eth_hdr(skb);
if (unlikely(is_multicast_ether_addr(eth->h_dest))) {
^ permalink raw reply related
* RE: [PATCH 1/1] net/usb: initiate sync sequence in sierra_net.c driver
From: Elina Pasheva @ 2010-05-02 5:53 UTC (permalink / raw)
To: David Miller
Cc: dbrownell@users.sourceforge.net, Rory Filer,
linux-usb@vger.kernel.org, netdev@vger.kernel.org
In-Reply-To: <20100501.180829.139101312.davem@davemloft.net>
> On Saturday, May 01, 2010 6:08 PM David Miller wrote:
>>From: Elina Pasheva <epasheva@sierrawireless.com>
>>Date: Wed, 28 Apr 2010 16:28:24 -0700
>> Subject: [PATCH 1/1] net/usb: initiate sync sequence in sierra_net.c driver
>> From: Elina Pasheva <epasheva@sierrawireless.com>
>>
>> The following patch adds the initiation of the sync sequence to
>> "sierra_net_bind()". If this step is omitted, the modem will never sync up
>> with the host and it will not be possible to establish a data connection.
>> This is a high priority patch.
>>
>> This patch has been checked against net-2.6 tree.
>> Signed-off-by: Elina Pasheva <epasheva@sierrawireless.com>
>> Signed-off-by: Rory Filer <rfiler@sierrawireless.com>
>> Tested-by: Elina Pasheva <epasheva@sierrawireless.com>
>Applied.
Thank you very much, David!
Elina
^ permalink raw reply
* Re: [PATCH net-next-2.6] net: eth_type_trans() should inline skb_pull()
From: Eric Dumazet @ 2010-05-02 6:50 UTC (permalink / raw)
To: David Miller; +Cc: netdev, therbert, hadi
In-Reply-To: <20100501.181558.141243424.davem@davemloft.net>
Le samedi 01 mai 2010 à 18:15 -0700, David Miller a écrit :
> From: Eric Dumazet <eric.dumazet@gmail.com>
> Date: Sat, 01 May 2010 08:42:25 +0200
>
> > [PATCH net-next-2.6] net: eth_type_trans() should inline skb_pull()
> >
> > With RPS, this patch can give a 5 % boost in performance.
> >
> > Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
>
> Awesome, but let's do this in a way that allows us to easily annotate
> where inlining makes sense in other places, not just here.
>
> Something like this, ok?
>
> diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
> index 82f5116..746a652 100644
> --- a/include/linux/skbuff.h
> +++ b/include/linux/skbuff.h
> @@ -1128,6 +1128,11 @@ static inline unsigned char *__skb_pull(struct sk_buff *skb, unsigned int len)
> return skb->data += len;
> }
>
> +static inline unsigned char *skb_pull_inline(struct sk_buff *skb, unsigned int len)
> +{
> + return unlikely(len > skb->len) ? NULL : __skb_pull(skb, len);
> +}
> +
> extern unsigned char *__pskb_pull_tail(struct sk_buff *skb, int delta);
>
> static inline unsigned char *__pskb_pull(struct sk_buff *skb, unsigned int len)
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index 4218ff4..8b9c109 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -1051,7 +1051,7 @@ EXPORT_SYMBOL(skb_push);
> */
> unsigned char *skb_pull(struct sk_buff *skb, unsigned int len)
> {
> - return unlikely(len > skb->len) ? NULL : __skb_pull(skb, len);
> + return skb_pull_inline(skb, len);
> }
> EXPORT_SYMBOL(skb_pull);
>
> diff --git a/net/ethernet/eth.c b/net/ethernet/eth.c
> index 0c0d272..61ec032 100644
> --- a/net/ethernet/eth.c
> +++ b/net/ethernet/eth.c
> @@ -162,7 +162,7 @@ __be16 eth_type_trans(struct sk_buff *skb, struct net_device *dev)
>
> skb->dev = dev;
> skb_reset_mac_header(skb);
> - skb_pull(skb, ETH_HLEN);
> + skb_pull_inline(skb, ETH_HLEN);
> eth = eth_hdr(skb);
>
> if (unlikely(is_multicast_ether_addr(eth->h_dest))) {
Excellent !
Changli privately asked me why we were ignoring cases where skb->len <
ETH_HLEN.
I replied that minimum frame size was 46+12, then he asked me why we
were testing another time :
if (skb->len >= 2 && *(unsigned short *)rawp == 0xFFFF)
return htons(ETH_P_802_3);
Could we assume all eth_type_trans() must call it with initial skb->len
>= (46 + 12) or not ?
(According to ethernet specs, all frames should have a minimum payload
of 46 bytes)
If not sure, maybe we should issue a WARN_ON_ONCE()
If yes, tests could be removed and we could gain two cycles ;)
^ permalink raw reply
* Re: [PATCH v6] net: batch skb dequeueing from softnet input_pkt_queue
From: Eric Dumazet @ 2010-05-02 6:56 UTC (permalink / raw)
To: Andi Kleen
Cc: David Miller, hadi, xiaosuo, therbert, shemminger, netdev, lenb,
arjan
In-Reply-To: <20100501110000.GB9434@gargoyle.fritz.box>
Le samedi 01 mai 2010 à 13:00 +0200, Andi Kleen a écrit :
> On Fri, Apr 30, 2010 at 04:38:57PM -0700, David Miller wrote:
> > From: Andi Kleen <ak@gargoyle.fritz.box>
> > Date: Thu, 29 Apr 2010 23:41:44 +0200
> >
> > > Use io_schedule() in network stack to tell cpuidle governour to guarantee lower latencies
> > >
> > > XXX: probably too aggressive, some of these sleeps are not under high load.
> > >
> > > Based on a bug report from Eric Dumazet.
> > >
> > > Signed-off-by: Andi Kleen <ak@linux.intel.com>
> >
> > I like this, except that we probably don't want the delayacct_blkio_*() calls
> > these things do.
>
> Yes.
>
> It needs more work, please don't apply it yet, to handle the "long sleep" case.
>
> Still curious if it fixes Eric's test case.
>
I tried it on the right spot (since my bench was only doing recvmsg()
calls, I had to patch wait_for_packet() in net/core/datagram.c
udp_recvmsg -> __skb_recv_datagram -> wait_for_packet ->
schedule_timeout
Unfortunatly, using io_schedule_timeout() did not solve the problem.
Tell me if you need some traces or something.
Thanks !
diff --git a/net/core/datagram.c b/net/core/datagram.c
index 95b851f..051fd5b 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -113,7 +113,7 @@ static int wait_for_packet(struct sock *sk, int *err, long *timeo_p)
goto interrupted;
error = 0;
- *timeo_p = schedule_timeout(*timeo_p);
+ *timeo_p = io_schedule_timeout(*timeo_p);
out:
finish_wait(sk_sleep(sk), &wait);
return error;
^ permalink raw reply related
* Re: [PATCH v6] net: batch skb dequeueing from softnet input_pkt_queue
From: Andi Kleen @ 2010-05-02 9:20 UTC (permalink / raw)
To: Eric Dumazet
Cc: David Miller, hadi, xiaosuo, therbert, shemminger, netdev, lenb,
arjan
In-Reply-To: <1272783366.2173.13.camel@edumazet-laptop>
> I tried it on the right spot (since my bench was only doing recvmsg()
> calls, I had to patch wait_for_packet() in net/core/datagram.c
>
> udp_recvmsg -> __skb_recv_datagram -> wait_for_packet ->
> schedule_timeout
>
> Unfortunatly, using io_schedule_timeout() did not solve the problem.
Hmm, too bad. Weird.
>
> Tell me if you need some traces or something.
I'll try to reproduce it and see what I can do.
-Andi
^ permalink raw reply
* Re: [PATCH net-next-2.6] net: eth_type_trans() should inline skb_pull()
From: David Miller @ 2010-05-02 10:03 UTC (permalink / raw)
To: eric.dumazet; +Cc: netdev, therbert, hadi
In-Reply-To: <1272783032.2173.8.camel@edumazet-laptop>
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Sun, 02 May 2010 08:50:32 +0200
> Excellent !
Great, here is the commit message I will use:
--------------------
net: Inline skb_pull() in eth_type_trans().
In commit 6be8ac2f ("[NET]: uninline skb_pull, de-bloats a lot")
we uninlined skb_pull.
But in some critical paths it makes sense to inline this thing
and it helps performance significantly.
Create an skb_pull_inline() so that we can do this in a way that
serves also as annotation.
Based upon a patch by Eric Dumazet.
Signed-off-by: David S. Miller <davem@davemloft.net>
--------------------
> Could we assume all eth_type_trans() must call it with initial
> skb->len >= (46 + 12) or not ? (According to ethernet specs, all
> frames should have a minimum payload of 46 bytes)
>
> If not sure, maybe we should issue a WARN_ON_ONCE()
>
> If yes, tests could be removed and we could gain two cycles ;)
Isn't the minimum ETH_ZLEN?
But yes, regardless of whether the minimum ethernet frame is 58 or 60
bytes, we should require it's at least that big, and use that test
consistently throughout.
Anything smaller is a runt packet and should be tossed or marked as an
error in some other way by the hardware. They should never make it to
eth_type_trans().
^ permalink raw reply
* Re: [PATCH 1/3] ptp: Added a brand new class driver for ptp clocks.
From: Wolfgang Grandegger @ 2010-05-02 10:50 UTC (permalink / raw)
To: Richard Cochran; +Cc: netdev
In-Reply-To: <20100429091936.GA6703@riccoc20.at.omicron.at>
Hi Richard,
Richard Cochran wrote:
> This patch adds an infrastructure for hardware clocks that implement
> IEEE 1588, the Precision Time Protocol (PTP). A class driver offers a
> registration method to particular hardware clock drivers. Each clock is
> exposed to user space as a character device with ioctls that allow tuning
> of the PTP clock.
>
> Signed-off-by: Richard Cochran <richard.cochran@omicron.at>
> ---
...
> diff --git a/drivers/ptp/ptp_clock.c b/drivers/ptp/ptp_clock.c
> new file mode 100644
> index 0000000..a5acac4
> --- /dev/null
> +++ b/drivers/ptp/ptp_clock.c
...
> +static int ptp_open(struct inode *inode, struct file *fp)
> +{
> + struct ptp_clock *ptp;
> + ptp = container_of(inode->i_cdev, struct ptp_clock, cdev);
> +
> + if (mutex_lock_interruptible(&ptp->mux))
> + return -ERESTARTSYS;
> +
> + fp->private_data = ptp;
> +
> + return 0;
> +}
...
> +static int ptp_release(struct inode *inode, struct file *fp)
> +{
> + struct ptp_clock *ptp = fp->private_data;
> + mutex_unlock(&ptp->mux);
> + return 0;
> +}
As long as the device is in use by an application, no other can access
it, because the mutex is locked. Other application may want to read the
PTP clock time while ptpd is running, though.
Wolfgang.
^ permalink raw reply
* Re: [PATCH v6] net: batch skb dequeueing from softnet input_pkt_queue
From: Eric Dumazet @ 2010-05-02 10:54 UTC (permalink / raw)
To: Andi Kleen
Cc: David Miller, hadi, xiaosuo, therbert, shemminger, netdev, lenb,
arjan
In-Reply-To: <20100502092020.GA9655@gargoyle.fritz.box>
Le dimanche 02 mai 2010 à 11:20 +0200, Andi Kleen a écrit :
> > I tried it on the right spot (since my bench was only doing recvmsg()
> > calls, I had to patch wait_for_packet() in net/core/datagram.c
> >
> > udp_recvmsg -> __skb_recv_datagram -> wait_for_packet ->
> > schedule_timeout
> >
> > Unfortunatly, using io_schedule_timeout() did not solve the problem.
>
> Hmm, too bad. Weird.
>
> >
> > Tell me if you need some traces or something.
>
> I'll try to reproduce it and see what I can do.
>
Here the perf report on the latest test done, I confirm I am using
io_schedule_timeout() in this kernel.
In this test, all 16 queues of one BCM57711E NIC (1Gb link) delivers
packets at about 1.300.000 pps to 16 cpus (one cpu per queue) and these
packets are then redistributed by RPS to same 16 cpus, generating about
650.000 IPI per second.
top says :
Cpu(s): 3.0%us, 17.3%sy, 0.0%ni, 22.4%id, 28.2%wa, 0.0%hi, 29.1%si,
0.0%st
# Samples: 321362570767
#
# Overhead Command Shared Object Symbol
# ........ .............. ............................ ......
#
25.08% init [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
|
|--93.47%-- clockevents_notify
| lapic_timer_state_broadcast
| acpi_idle_enter_bm
| cpuidle_idle_call
| cpu_idle
| start_secondary
|
|--4.70%-- tick_broadcast_oneshot_control
| tick_notify
| notifier_call_chain
| __raw_notifier_call_chain
| raw_notifier_call_chain
| clockevents_do_notify
| clockevents_notify
| lapic_timer_state_broadcast
| acpi_idle_enter_bm
| cpuidle_idle_call
| cpu_idle
| start_secondary
|
|--0.64%-- generic_exec_single
| __smp_call_function_single
| net_rps_action_and_irq_enable
...
9.72% init [kernel.kallsyms] [k] acpi_os_read_port
|
--- acpi_os_read_port
|
|--99.45%-- acpi_hw_read_port
| acpi_hw_read
| acpi_hw_read_multiple
| acpi_hw_register_read
| acpi_read_bit_register
| acpi_idle_enter_bm
| cpuidle_idle_call
| cpu_idle
| start_secondary
|
--0.55%-- acpi_hw_read
acpi_hw_read_multiple
powertop says :
PowerTOP version 1.11 (C) 2007 Intel Corporation
Cn Avg residency P-states (frequencies)
C0 (cpu running) (68.9%) 2.93 Ghz 46.5%
polling 0.0ms ( 0.0%) 2.80 Ghz 5.1%
C1 mwait 0.0ms ( 0.0%) 2.53 Ghz 3.0%
C2 mwait 0.0ms (31.1%) 2.13 Ghz 2.8%
1.60 Ghz 38.2%
Wakeups-from-idle per second : 45177.8 interval: 5.0s
no ACPI power usage estimate available
Top causes for wakeups:
9.9% (40863.0) <interrupt> : eth1-fp-7
9.9% (40861.0) <interrupt> : eth1-fp-8
9.9% (40858.0) <interrupt> : eth1-fp-5
9.9% (40855.2) <interrupt> : eth1-fp-10
9.9% (40847.6) <interrupt> : eth1-fp-14
9.9% (40847.2) <interrupt> : eth1-fp-12
9.9% (40835.0) <interrupt> : eth1-fp-1
9.9% (40834.2) <interrupt> : eth1-fp-3
9.9% (40834.0) <interrupt> : eth1-fp-6
9.9% (40829.6) <interrupt> : eth1-fp-4
1.0% (4002.0) <kernel core> : hrtimer_start_range_ns (tick_sched_timer)
0.4% (1725.6) <interrupt> : extra timer interrupt
0.0% ( 4.0) <kernel core> : usb_hcd_poll_rh_status (rh_timer_func)
0.0% ( 2.0) <kernel core> : clocksource_watchdog (clocksource_watchdog)
0.0% ( 2.0) snmpd : hrtimer_start_range_ns (hrtimer_wakeup)
^ permalink raw reply
* Re: Re: Kernel panic in fib_rules_lookup (kernel 2.6.32)
From: "Oleg A. Arkhangelsky" @ 2010-05-02 10:46 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev
In-Reply-To: <1268154568.3113.19.camel@edumazet-laptop>
Hello,
09.03.10, 17:09, "Eric Dumazet" <eric.dumazet@gmail.com>:
> Le mardi 09 mars 2010 à 10:44 +0300, "Oleg A. Arkhangelsky" a écrit :
> > Hello,
> >
> > Got this kernel panic tomorrow. This PC is rather heavy loaded router with BGP full view (> 300K routes).
> > We're using FIB_TRIE. Last time we got similar panic about 1 month ago. Please, let me know if you
> > need additional information to debug (e.g. objdump). Thanks!
> >
> > Mar 9 10:08:55 bras-1 kernel: BUG: unable to handle kernel NULL pointer dereference at (null)
> > Mar 9 10:08:55 bras-1 kernel: IP: [] fib_rules_lookup+0xa2/0xd0
> > Mar 9 10:08:55 bras-1 kernel: *pde = 00000000
> > Mar 9 10:08:55 bras-1 kernel: Thread overran stack, or stack corrupted
>
> Hmm...
Got the same panic, at the same place (fib_rules_lookup+0xa2/0xd0). Looks like the problem with NULL dereference is somewhere in list_for_each_entry_rcu macro. But I don't understand how this can be.
Any thoughts? :(
--
wbr, Oleg.
^ permalink raw reply
* Re: ep93xx_eth stopps receiving packages
From: Lennert Buytenhek @ 2010-05-02 10:43 UTC (permalink / raw)
To: Stefan Agner; +Cc: netdev
In-Reply-To: <20100419173813.7750395f4fkkmrk0@limpopo.deheime.ch>
On Mon, Apr 19, 2010 at 05:38:13PM +0200, Stefan Agner wrote:
> I'm using Linux 2.6.32.9 on a technologic systems TS-7250 SBC board, with
> the ep93xx_eth driver for networking. On three identical, but independent
> systems I noted that the system is unreachable after a while. On a serial
> terminal I noted that only the TX counter counts onward, RX stays where it
> is,
> no matter if i try to ping from or to the system. Wireshark tells me exactly
> that too: I see helpless ARP requests which gets answered, but no ICMP. The
> system doesnt receive the ARP requests, and just sends another one.
(So does the board or does it not respond to ARP requests for its IP?)
> With a simple program which sends small packages in a fast pace I can
> reproduce the problem after several seconds (additional CPU load seem to
> provoke the problem even more). Remove and replug the network cable doesn't
> solve the problem, but ifup/down does. I don't see any messages in dmesg,
> memory is still available.
Do you see interrupts increasing in /proc/interrupts when this happens?
^ permalink raw reply
* Re: [PATCH 0/3] [RFC] ptp: IEEE 1588 clock support
From: Wolfgang Grandegger @ 2010-05-02 11:51 UTC (permalink / raw)
To: Richard Cochran; +Cc: netdev
In-Reply-To: <20100429153401.GA26741@riccoc20.at.omicron.at>
Richard Cochran wrote:
> On Thu, Apr 29, 2010 at 02:02:59PM +0200, Wolfgang Grandegger wrote:
>> I realized two other netdev drivers already supporting PTP timestamping:
>> igb and bfin_mac. From the PTP developer point of view, the interface
>> looks rather complete to me and it works fine on my MPC8313 setup.
>
> Do you know whether these two also have PTP clocks? If so, is the API
> that I suggested going to work for controlling those clocks, too?
>
>> The only thing I stumbled over was that PTP clock registration
>> failed when PTP support is statically linked into the kernel.
>
> Okay, will look into that...
With subsys_initcall(), ptp_gianfar_init() is called very early and
*before* ptp_init(). It works fine with module_init(). The
ptp_gianfar_init() is then called after gianfar_init().
There is another minor issue with module init/probe ordering.
gianfar_ptp_probe() does overtake the time from the system clock, which
may happen before the RTC is probed and initialized. But syncing with
the system time is a separate issue anyway.
Wolfgang.
^ permalink raw reply
* Re: [PATCH v6] net: batch skb dequeueing from softnet input_pkt_queue
From: Arjan van de Ven @ 2010-05-02 14:13 UTC (permalink / raw)
To: Eric Dumazet
Cc: Andi Kleen, David Miller, hadi, xiaosuo, therbert, shemminger,
netdev, lenb
In-Reply-To: <1272797690.2173.26.camel@edumazet-laptop>
>
> Cn Avg residency P-states (frequencies)
> C0 (cpu running) (68.9%) 2.93 Ghz 46.5%
> polling 0.0ms ( 0.0%) 2.80 Ghz 5.1%
> C1 mwait 0.0ms ( 0.0%) 2.53 Ghz 3.0%
> C2 mwait 0.0ms (31.1%) 2.13 Ghz 2.8%
> 1.60 Ghz 38.2%
I bet your system advertizes C2 with the same latency as C1,
but with lower power... which means Linux will pretty much never
pick C1.... no matter how much you take Andi's patch.
this is a bios thing... and until we put in the patch to override the
bios values (I can dust it off but it might need a bit of tweaking
since it was against .31) Andi's patch alone won't cut it... you also
need a non-lying bios ;)
--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
^ permalink raw reply
* Re: [PATCH v6] net: batch skb dequeueing from softnet input_pkt_queue
From: Eric Dumazet @ 2010-05-02 14:27 UTC (permalink / raw)
To: Arjan van de Ven
Cc: Andi Kleen, David Miller, hadi, xiaosuo, therbert, shemminger,
netdev, lenb
In-Reply-To: <20100502071324.1a95fdad@infradead.org>
Le dimanche 02 mai 2010 à 07:13 -0700, Arjan van de Ven a écrit :
> >
> > Cn Avg residency P-states (frequencies)
> > C0 (cpu running) (68.9%) 2.93 Ghz 46.5%
> > polling 0.0ms ( 0.0%) 2.80 Ghz 5.1%
> > C1 mwait 0.0ms ( 0.0%) 2.53 Ghz 3.0%
> > C2 mwait 0.0ms (31.1%) 2.13 Ghz 2.8%
> > 1.60 Ghz 38.2%
>
> I bet your system advertizes C2 with the same latency as C1,
> but with lower power... which means Linux will pretty much never
> pick C1.... no matter how much you take Andi's patch.
>
> this is a bios thing... and until we put in the patch to override the
> bios values (I can dust it off but it might need a bit of tweaking
> since it was against .31) Andi's patch alone won't cut it... you also
> need a non-lying bios ;)
>
>
>
# pwd
/sys/devices/system/cpu/cpu15/cpuidle
# grep . */*
state0/desc:CPUIDLE CORE POLL IDLE
state0/latency:0
state0/name:C0
state0/power:4294967295
state0/time:0
state0/usage:0
state1/desc:ACPI FFH INTEL MWAIT 0x0
state1/latency:1
state1/name:C1
state1/power:1000
state1/time:433855186
state1/usage:126869
state2/desc:ACPI FFH INTEL MWAIT 0x10
state2/latency:64
state2/name:C2
state2/power:500
state2/time:198095020416
state2/usage:76287744
C2 latency seems to be 64 (us ?), while C1 seems to be 1
BIOS Information
Vendor: HP
Version: I24
Release Date: 10/01/2009
# powertop
PowerTOP 1.11 (C) 2007, 2008 Intel Corporation
Collecting data for 5 seconds
Your CPU supports the following C-states : C1 C2 C3
Your BIOS reports the following C-states : C1 C2
C3 seems to be disabled in BIOS
^ permalink raw reply
* Re: [PATCH v6] net: batch skb dequeueing from softnet input_pkt_queue
From: Eric Dumazet @ 2010-05-02 15:32 UTC (permalink / raw)
To: Arjan van de Ven
Cc: Andi Kleen, David Miller, hadi, xiaosuo, therbert, shemminger,
netdev, lenb
In-Reply-To: <1272810448.2173.31.camel@edumazet-laptop>
Le dimanche 02 mai 2010 à 16:27 +0200, Eric Dumazet a écrit :
> Le dimanche 02 mai 2010 à 07:13 -0700, Arjan van de Ven a écrit :
> > >
> > > Cn Avg residency P-states (frequencies)
> > > C0 (cpu running) (68.9%) 2.93 Ghz 46.5%
> > > polling 0.0ms ( 0.0%) 2.80 Ghz 5.1%
> > > C1 mwait 0.0ms ( 0.0%) 2.53 Ghz 3.0%
> > > C2 mwait 0.0ms (31.1%) 2.13 Ghz 2.8%
> > > 1.60 Ghz 38.2%
> >
> > I bet your system advertizes C2 with the same latency as C1,
> > but with lower power... which means Linux will pretty much never
> > pick C1.... no matter how much you take Andi's patch.
> >
> > this is a bios thing... and until we put in the patch to override the
> > bios values (I can dust it off but it might need a bit of tweaking
> > since it was against .31) Andi's patch alone won't cut it... you also
> > need a non-lying bios ;)
> >
> >
> >
> # pwd
> /sys/devices/system/cpu/cpu15/cpuidle
> # grep . */*
> state0/desc:CPUIDLE CORE POLL IDLE
> state0/latency:0
> state0/name:C0
> state0/power:4294967295
> state0/time:0
> state0/usage:0
> state1/desc:ACPI FFH INTEL MWAIT 0x0
> state1/latency:1
> state1/name:C1
> state1/power:1000
> state1/time:433855186
> state1/usage:126869
> state2/desc:ACPI FFH INTEL MWAIT 0x10
> state2/latency:64
> state2/name:C2
> state2/power:500
> state2/time:198095020416
> state2/usage:76287744
>
> C2 latency seems to be 64 (us ?), while C1 seems to be 1
>
> BIOS Information
> Vendor: HP
> Version: I24
> Release Date: 10/01/2009
>
> # powertop
> PowerTOP 1.11 (C) 2007, 2008 Intel Corporation
>
> Collecting data for 5 seconds
>
>
> Your CPU supports the following C-states : C1 C2 C3
> Your BIOS reports the following C-states : C1 C2
>
> C3 seems to be disabled in BIOS
>
I took a look at BIOS settings and enabled the minimum sleep state to be
C6 (instead of C3, the default). Now we see C3 being available...
No changes, only more IPI delivered during the test, and more overhead
in clockevents_notify()
# grep . */*
state0/desc:CPUIDLE CORE POLL IDLE
state0/latency:0
state0/name:C0
state0/power:4294967295
state0/time:0
state0/usage:0
state1/desc:ACPI FFH INTEL MWAIT 0x0
state1/latency:1
state1/name:C1
state1/power:1000
state1/time:39432
state1/usage:119
state2/desc:ACPI FFH INTEL MWAIT 0x10
state2/latency:64
state2/name:C2
state2/power:500
state2/time:3170745
state2/usage:11177
state3/desc:ACPI FFH INTEL MWAIT 0x20
state3/latency:96
state3/name:C3
state3/power:350
state3/time:1030987453
state3/usage:14047019
---------------------------------------------------------------------------------------------------------------------------
PerfTop: 15984 irqs/sec kernel:98.5% [1000Hz cycles], (all, 16 CPUs)
---------------------------------------------------------------------------------------------------------------------------
samples pcnt function DSO
_______ _____ ______________________________ _______
23822.00 40.2% _raw_spin_lock_irqsave vmlinux
4413.00 7.4% acpi_os_read_port vmlinux
1426.00 2.4% _raw_spin_lock vmlinux
1284.00 2.2% _raw_spin_unlock_irqrestore vmlinux
1247.00 2.1% schedule vmlinux
1137.00 1.9% bnx2x_rx_int vmlinux
643.00 1.1% tick_broadcast_oneshot_control vmlinux
597.00 1.0% copy_user_generic_string vmlinux
595.00 1.0% __napi_complete vmlinux
550.00 0.9% call_function_single_interrupt vmlinux
548.00 0.9% bnx2x_msix_fp_int vmlinux
486.00 0.8% __netif_receive_skb vmlinux
461.00 0.8% bnx2x_poll vmlinux
433.00 0.7% eth_type_trans vmlinux
428.00 0.7% acpi_idle_enter_bm vmlinux
422.00 0.7% sock_recv_ts_and_drops vmlinux
382.00 0.6% __udp4_lib_lookup vmlinux
369.00 0.6% __slab_free vmlinux
357.00 0.6% ip_route_input vmlinux
341.00 0.6% kfree vmlinux
335.00 0.6% ipt_do_table vmlinux
334.00 0.6% ip_rcv vmlinux
332.00 0.6% udp_recvmsg vmlinux
317.00 0.5% __kmalloc_node_track_caller vmlinux
37.46% init [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
|
|--95.58%-- clockevents_notify
| lapic_timer_state_broadcast
| acpi_idle_enter_bm
| cpuidle_idle_call
| cpu_idle
| start_secondary
|
|--3.27%-- tick_broadcast_oneshot_control
| tick_notify
| notifier_call_chain
| __raw_notifier_call_chain
| raw_notifier_call_chain
| clockevents_do_notify
| clockevents_notify
| lapic_timer_state_broadcast
| acpi_idle_enter_bm
^ permalink raw reply
* [PATCH v3] net: fix softnet_stat
From: Changli Gao @ 2010-05-02 15:42 UTC (permalink / raw)
To: David S. Miller; +Cc: Eric Dumazet, netdev, Changli Gao
fix softnet_stat
Per cpu variable softnet_data.total was shared between IRQ and SoftIRQ context
without any protection. And enqueue_to_backlog should update the netdev_rx_stat
of the target CPU.
This patch renames softnet_data.total to softnet_data.processed: the number of
packets processed in uppper levels(IP stacks).
softnet_stat data is moved into softnet_data.
Signed-off-by: Changli Gao <xiaosuo@gmail.com>
----
include/linux/netdevice.h | 17 +++++++----------
net/core/dev.c | 26 ++++++++++++--------------
net/sched/sch_generic.c | 2 +-
3 files changed, 20 insertions(+), 25 deletions(-)
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 40d4c20..c39938f 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -218,16 +218,6 @@ struct neighbour;
struct neigh_parms;
struct sk_buff;
-struct netif_rx_stats {
- unsigned total;
- unsigned dropped;
- unsigned time_squeeze;
- unsigned cpu_collision;
- unsigned received_rps;
-};
-
-DECLARE_PER_CPU(struct netif_rx_stats, netdev_rx_stat);
-
struct netdev_hw_addr {
struct list_head list;
unsigned char addr[MAX_ADDR_LEN];
@@ -1390,6 +1380,12 @@ struct softnet_data {
struct sk_buff *completion_queue;
struct sk_buff_head process_queue;
+ /* stats */
+ unsigned processed;
+ unsigned time_squeeze;
+ unsigned cpu_collision;
+ unsigned received_rps;
+
#ifdef CONFIG_RPS
struct softnet_data *rps_ipi_list;
@@ -1399,6 +1395,7 @@ struct softnet_data {
unsigned int cpu;
unsigned int input_queue_head;
#endif
+ unsigned dropped;
struct sk_buff_head input_pkt_queue;
struct napi_struct backlog;
};
diff --git a/net/core/dev.c b/net/core/dev.c
index 100dcbd..36d53be 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -2205,8 +2205,6 @@ int netdev_max_backlog __read_mostly = 1000;
int netdev_budget __read_mostly = 300;
int weight_p __read_mostly = 64; /* old backlog weight */
-DEFINE_PER_CPU(struct netif_rx_stats, netdev_rx_stat) = { 0, };
-
#ifdef CONFIG_RPS
/* One global table that all flow-based protocols share. */
@@ -2366,7 +2364,7 @@ static void rps_trigger_softirq(void *data)
struct softnet_data *sd = data;
__napi_schedule(&sd->backlog);
- __get_cpu_var(netdev_rx_stat).received_rps++;
+ sd->received_rps++;
}
#endif /* CONFIG_RPS */
@@ -2405,7 +2403,6 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
sd = &per_cpu(softnet_data, cpu);
local_irq_save(flags);
- __get_cpu_var(netdev_rx_stat).total++;
rps_lock(sd);
if (skb_queue_len(&sd->input_pkt_queue) <= netdev_max_backlog) {
@@ -2429,9 +2426,9 @@ enqueue:
goto enqueue;
}
+ sd->dropped++;
rps_unlock(sd);
- __get_cpu_var(netdev_rx_stat).dropped++;
local_irq_restore(flags);
kfree_skb(skb);
@@ -2806,7 +2803,7 @@ static int __netif_receive_skb(struct sk_buff *skb)
skb->dev = master;
}
- __get_cpu_var(netdev_rx_stat).total++;
+ __get_cpu_var(softnet_data).processed++;
skb_reset_network_header(skb);
skb_reset_transport_header(skb);
@@ -3490,7 +3487,7 @@ out:
return;
softnet_break:
- __get_cpu_var(netdev_rx_stat).time_squeeze++;
+ sd->time_squeeze++;
__raise_softirq_irqoff(NET_RX_SOFTIRQ);
goto out;
}
@@ -3691,17 +3688,17 @@ static int dev_seq_show(struct seq_file *seq, void *v)
return 0;
}
-static struct netif_rx_stats *softnet_get_online(loff_t *pos)
+static struct softnet_data *softnet_get_online(loff_t *pos)
{
- struct netif_rx_stats *rc = NULL;
+ struct softnet_data *sd = NULL;
while (*pos < nr_cpu_ids)
if (cpu_online(*pos)) {
- rc = &per_cpu(netdev_rx_stat, *pos);
+ sd = &per_cpu(softnet_data, *pos);
break;
} else
++*pos;
- return rc;
+ return sd;
}
static void *softnet_seq_start(struct seq_file *seq, loff_t *pos)
@@ -3721,12 +3718,12 @@ static void softnet_seq_stop(struct seq_file *seq, void *v)
static int softnet_seq_show(struct seq_file *seq, void *v)
{
- struct netif_rx_stats *s = v;
+ struct softnet_data *sd = v;
seq_printf(seq, "%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n",
- s->total, s->dropped, s->time_squeeze, 0,
+ sd->processed, sd->dropped, sd->time_squeeze, 0,
0, 0, 0, 0, /* was fastroute */
- s->cpu_collision, s->received_rps);
+ sd->cpu_collision, sd->received_rps);
return 0;
}
@@ -5869,6 +5866,7 @@ static int __init net_dev_init(void)
for_each_possible_cpu(i) {
struct softnet_data *sd = &per_cpu(softnet_data, i);
+ memset(sd, 0, sizeof(*sd));
skb_queue_head_init(&sd->input_pkt_queue);
skb_queue_head_init(&sd->process_queue);
sd->completion_queue = NULL;
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index aeddabf..a969b11 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -94,7 +94,7 @@ static inline int handle_dev_cpu_collision(struct sk_buff *skb,
* Another cpu is holding lock, requeue & delay xmits for
* some time.
*/
- __get_cpu_var(netdev_rx_stat).cpu_collision++;
+ __get_cpu_var(softnet_data).cpu_collision++;
ret = dev_requeue_skb(skb, q);
}
^ permalink raw reply related
* Re: [PATCH v6] net: batch skb dequeueing from softnet input_pkt_queue
From: Andi Kleen @ 2010-05-02 15:46 UTC (permalink / raw)
To: Eric Dumazet
Cc: David Miller, hadi, xiaosuo, therbert, shemminger, netdev, lenb,
arjan
In-Reply-To: <1272797690.2173.26.camel@edumazet-laptop>
> In this test, all 16 queues of one BCM57711E NIC (1Gb link) delivers
> packets at about 1.300.000 pps to 16 cpus (one cpu per queue) and these
> packets are then redistributed by RPS to same 16 cpus, generating about
> 650.000 IPI per second.
BTW if rps was SMT aware it could avoid a lot of the IPIs in the first place.
-Andi
^ permalink raw reply
* Re: Re: Kernel panic in fib_rules_lookup (kernel 2.6.32)
From: Eric Dumazet @ 2010-05-02 16:31 UTC (permalink / raw)
To: "Oleg A. Arkhangelsky"; +Cc: netdev
In-Reply-To: <382091272797186@web84.yandex.ru>
Le dimanche 02 mai 2010 à 14:46 +0400, "Oleg A. Arkhangelsky" a écrit :
> Hello,
>
> 09.03.10, 17:09, "Eric Dumazet" <eric.dumazet@gmail.com>:
>
> > Le mardi 09 mars 2010 à 10:44 +0300, "Oleg A. Arkhangelsky" a écrit :
> > > Hello,
> > >
> > > Got this kernel panic tomorrow. This PC is rather heavy loaded router with BGP full view (> 300K routes).
> > > We're using FIB_TRIE. Last time we got similar panic about 1 month ago. Please, let me know if you
> > > need additional information to debug (e.g. objdump). Thanks!
> > >
> > > Mar 9 10:08:55 bras-1 kernel: BUG: unable to handle kernel NULL pointer dereference at (null)
> > > Mar 9 10:08:55 bras-1 kernel: IP: [] fib_rules_lookup+0xa2/0xd0
> > > Mar 9 10:08:55 bras-1 kernel: *pde = 00000000
> > > Mar 9 10:08:55 bras-1 kernel: Thread overran stack, or stack corrupted
> >
> > Hmm...
>
> Got the same panic, at the same place (fib_rules_lookup+0xa2/0xd0). Looks like the problem with NULL dereference is somewhere in list_for_each_entry_rcu macro. But I don't understand how this can be.
>
> Any thoughts? :(
>
Do you have any modify rules activity ?
I dont understand why we need synchronize_rcu() in fib_nl_delrule() but
this certainly not a bug.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox