* Re: [REBASE PATCH net-next v9 0/4] net: vhost: improve performance when enable busyloop
From: David Miller @ 2018-09-27 3:26 UTC (permalink / raw)
To: xiangxia.m.yue; +Cc: jasowang, mst, makita.toshiaki, netdev, virtualization
In-Reply-To: <1537879012-20859-1-git-send-email-xiangxia.m.yue@gmail.com>
From: xiangxia.m.yue@gmail.com
Date: Tue, 25 Sep 2018 05:36:48 -0700
> From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
>
> This patches improve the guest receive performance.
> On the handle_tx side, we poll the sock receive queue
> at the same time. handle_rx do that in the same way.
>
> For more performance report, see patch 4
Series applied, thank you.
^ permalink raw reply
* Re: [PATCH v2 net] net: phy: sfp: Fix unregistering of HWMON SFP device
From: David Miller @ 2018-09-27 3:25 UTC (permalink / raw)
To: andrew; +Cc: netdev, rmk+kernel, f.fainelli
In-Reply-To: <1537833000-23960-1-git-send-email-andrew@lunn.ch>
From: Andrew Lunn <andrew@lunn.ch>
Date: Tue, 25 Sep 2018 01:50:00 +0200
> A HWMON device is only registered is the SFP module supports the
> diagnostic page and is complient to SFF8472. Don't unconditionally
> unregister the hwmon device when the SFP module is remove, otherwise
> we access data structures which don't exist.
>
> Reported-by: Florian Fainelli <f.fainelli@gmail.com>
> Fixes: 1323061a018a ("net: phy: sfp: Add HWMON support for module sensors")
> Signed-off-by: Andrew Lunn <andrew@lunn.ch>
Applied.
^ permalink raw reply
* Re: [PATCH] qed: Avoid constant logical operation warning in qed_vf_pf_acquire
From: David Miller @ 2018-09-27 3:24 UTC (permalink / raw)
To: natechancellor; +Cc: Ariel.Elior, everest-linux-l2, netdev, linux-kernel
In-Reply-To: <20180924221703.26454-1-natechancellor@gmail.com>
From: Nathan Chancellor <natechancellor@gmail.com>
Date: Mon, 24 Sep 2018 15:17:03 -0700
> Clang warns when a constant is used in a boolean context as it thinks a
> bitwise operation may have been intended.
>
> drivers/net/ethernet/qlogic/qed/qed_vf.c:415:27: warning: use of logical
> '&&' with constant operand [-Wconstant-logical-operand]
> if (!p_iov->b_pre_fp_hsi &&
> ^
> drivers/net/ethernet/qlogic/qed/qed_vf.c:415:27: note: use '&' for a
> bitwise operation
> if (!p_iov->b_pre_fp_hsi &&
> ^~
> &
> drivers/net/ethernet/qlogic/qed/qed_vf.c:415:27: note: remove constant
> to silence this warning
> if (!p_iov->b_pre_fp_hsi &&
> ~^~
> 1 warning generated.
>
> This has been here since commit 1fe614d10f45 ("qed: Relax VF firmware
> requirements") and I am not entirely sure why since 0 isn't a special
> case. Just remove the statement causing Clang to warn since it isn't
> required.
>
> Link: https://github.com/ClangBuiltLinux/linux/issues/126
> Signed-off-by: Nathan Chancellor <natechancellor@gmail.com>
Applied.
^ permalink raw reply
* Re: [PATCH] qed: Avoid implicit enum conversion in qed_iwarp_parse_rx_pkt
From: David Miller @ 2018-09-27 3:24 UTC (permalink / raw)
To: natechancellor; +Cc: Ariel.Elior, everest-linux-l2, netdev, linux-kernel
In-Reply-To: <20180924214212.22522-1-natechancellor@gmail.com>
From: Nathan Chancellor <natechancellor@gmail.com>
Date: Mon, 24 Sep 2018 14:42:12 -0700
> Clang warns when one enumerated type is implicitly converted to another.
>
> drivers/net/ethernet/qlogic/qed/qed_iwarp.c:1713:25: warning: implicit
> conversion from enumeration type 'enum tcp_ip_version' to different
> enumeration type 'enum qed_tcp_ip_version' [-Wenum-conversion]
> cm_info->ip_version = TCP_IPV4;
> ~ ^~~~~~~~
> drivers/net/ethernet/qlogic/qed/qed_iwarp.c:1733:25: warning: implicit
> conversion from enumeration type 'enum tcp_ip_version' to different
> enumeration type 'enum qed_tcp_ip_version' [-Wenum-conversion]
> cm_info->ip_version = TCP_IPV6;
> ~ ^~~~~~~~
> 2 warnings generated.
>
> Use the appropriate values from the expected type, qed_tcp_ip_version:
>
> TCP_IPV4 = QED_TCP_IPV4 = 0
> TCP_IPV6 = QED_TCP_IPV6 = 1
>
> Link: https://github.com/ClangBuiltLinux/linux/issues/125
> Signed-off-by: Nathan Chancellor <natechancellor@gmail.com>
Applied.
^ permalink raw reply
* Re: [PATCH net] bonding: avoid possible dead-lock
From: David Miller @ 2018-09-27 3:24 UTC (permalink / raw)
To: mahesh; +Cc: j.vosburgh, andy, vfalico, netdev, edumazet, maheshb
In-Reply-To: <20180924214011.234426-1-mahesh@bandewar.net>
From: Mahesh Bandewar <mahesh@bandewar.net>
Date: Mon, 24 Sep 2018 14:40:11 -0700
> From: Mahesh Bandewar <maheshb@google.com>
>
> Syzkaller reported this on a slightly older kernel but it's still
> applicable to the current kernel -
...
> Signed-off-by: Mahesh Bandewar <maheshb@google.com>
Applied and queued up for -stable.
^ permalink raw reply
* Re: [PATCH net] bonding: pass link-local packets to bonding master also.
From: David Miller @ 2018-09-27 3:24 UTC (permalink / raw)
To: mahesh; +Cc: j.vosburgh, andy, vfalico, netdev, edumazet, maheshb
In-Reply-To: <20180924213942.234209-1-mahesh@bandewar.net>
From: Mahesh Bandewar <mahesh@bandewar.net>
Date: Mon, 24 Sep 2018 14:39:42 -0700
> From: Mahesh Bandewar <maheshb@google.com>
>
> Commit b89f04c61efe ("bonding: deliver link-local packets with
> skb->dev set to link that packets arrived on") changed the behavior
> of how link-local-multicast packets are processed. The change in
> the behavior broke some legacy use cases where these packets are
> expected to arrive on bonding master device also.
>
> This patch passes the packet to the stack with the link it arrived
> on as well as passes to the bonding-master device to preserve the
> legacy use case.
>
> Fixes: b89f04c61efe ("bonding: deliver link-local packets with skb->dev set to link that packets arrived on")
> Reported-by: Michal Soltys <soltys@ziu.info>
> Signed-off-by: Mahesh Bandewar <maheshb@google.com>
Applied and queued up for -stable.
^ permalink raw reply
* Re: [PATCH] net/ncsi: Add NCSI OEM command for FB Tiogapass
From: Joel Stanley @ 2018-09-27 3:20 UTC (permalink / raw)
To: Justin.Lee1
Cc: Amithash Prasad, vijaykhemka, Sam Mendoza-Jonas, linux-aspeed,
OpenBMC Maillist, sdasari, netdev, Christian Svensson
In-Reply-To: <5e9c8e3f8fd54845b15cd460efe9090d@AUSX13MPS306.AMER.DELL.COM>
On Thu, 27 Sep 2018 at 00:39, <Justin.Lee1@dell.com> wrote:
>
> > > As I understand Justin's version adds a generic handler, using the NCSI
> > > Netlink interface to pass OEM commands and responses to and from
> > > userspace, which does the actual packet handling.
> > Thanks for the direction Sam! Justin, if you don't mind, can you share the patches you have to add the support? This actually would solve a few other things we are trying to accomplish.
>
>
> Basically, I add a new flag to indicate the request is coming from the Netlink interface to allow the command handler and response handler to react.
> #define NCSI_REQ_FLAG_NETLINK_DRIVEN 2
>
> The work flow is as below.
>
> Request:
> User space application -> Netlink interface (msg) -> new Netlink handler - ncsi_send_cmd_nl() - ncsi_xmit_cmd()
>
> Response:
> Response received - ncsi_rcv_rsp() -> internal response handler - ncsi_rsp_handler_xxx() -> ncsi_send_netlink_rsp () -> Netlink interface (msg) -> user space application
> Command timeout - ncsi_request_timeout() -> ncsi_send_netlink_timeout () -> Netlink interface (msg with zero data length) -> user space application
>
> Error:
> Detected error -> ncsi_send_netlink_err () -> Netlink interface (err msg) -> user space application
>
> I will clean up some code and send out the patch.
Thanks for the overview. We look forward to your patches; please
include the same cc list as this series.
I think it makes sense to have some OEM NCSI handing purely in the
kenrel. This would allow eg. the MAC address of an interface to be
correct at boot, without requiring userspace to come up first.
I have also heard stories of temperature sensors over NCSI. Those
would make sense to be hwmon drivers, which again would mean handling
them in the kernel.
Justin, Vijay, can you please list the known NCSI OEM
commands/extensions that we plan on implementing?
Cheers,
Joel
^ permalink raw reply
* Re: [PATCH net-next] net: phy: improve handling delayed work
From: David Miller @ 2018-09-27 3:20 UTC (permalink / raw)
To: hkallweit1; +Cc: f.fainelli, andrew, netdev
In-Reply-To: <002b6ead-9d72-0995-4ac7-369a4422b026@gmail.com>
From: Heiner Kallweit <hkallweit1@gmail.com>
Date: Mon, 24 Sep 2018 23:36:50 +0200
> diff --git a/drivers/net/phy/phy.c b/drivers/net/phy/phy.c
> index a1f8e4816..40a7dc101 100644
> --- a/drivers/net/phy/phy.c
> +++ b/drivers/net/phy/phy.c
...
> @@ -635,6 +635,13 @@ int phy_speed_up(struct phy_device *phydev)
> }
> EXPORT_SYMBOL_GPL(phy_speed_up);
>
> +static inline void phy_queue_state_machine(struct phy_device *phydev,
> + unsigned int secs)
> +{
Please do not use the inline keyword in foo.c files, let the compiler
decide.
^ permalink raw reply
* Re: bond: take rcu lock in bond_poll_controller
From: David Miller @ 2018-09-27 3:15 UTC (permalink / raw)
To: davej; +Cc: netdev
In-Reply-To: <20180924192317.wghbs34ld7klqpi6@codemonkey.org.uk>
From: Dave Jones <davej@codemonkey.org.uk>
Date: Mon, 24 Sep 2018 15:23:17 -0400
> Callers of bond_for_each_slave_rcu are expected to hold the rcu lock,
> otherwise a trace like below is shown
...
> Signed-off-by: Dave Jones <davej@codemonkey.org.uk>
Hey Dave, after some recent changes by Eric Dumazet this no longer
applies.
Please respin against 'net'.
Thanks.
^ permalink raw reply
* Re: [PATCH v3 net-next 07/12] net: ethernet: Add helper to remove a supported link mode
From: David Miller @ 2018-09-27 3:08 UTC (permalink / raw)
To: andrew; +Cc: horms, netdev, f.fainelli, sergei.shtylyov, linux-renesas-soc
In-Reply-To: <20180924155023.GB10667@lunn.ch>
From: Andrew Lunn <andrew@lunn.ch>
Date: Mon, 24 Sep 2018 17:50:23 +0200
> I submitted it to netdev in the usual way. I hope DaveM will accept
> and merge it.
Andrew did I miss your patch somehow?
If so, sorry, please resend.
^ permalink raw reply
* Re: [PATCH net v2 0/2] net: phy: fix WoL handling when suspending the PHY
From: David Miller @ 2018-09-27 3:04 UTC (permalink / raw)
To: hkallweit1; +Cc: andrew, f.fainelli, nic_swsd, netdev
In-Reply-To: <43386295-806b-ff32-4c7c-56aaeb87b4db@gmail.com>
From: Heiner Kallweit <hkallweit1@gmail.com>
Date: Mon, 24 Sep 2018 21:58:04 +0200
> phy_suspend doesn't always recognize that WoL is enabled and therefore
> suspends the PHY when it should not. First idea to address the issue
> was to reuse checks used in mdio_bus_phy_may_suspend which check
> whether relevant devices are wakeup-enabled.
> Florian raised some concerns because drivers may enable wakeup even if
> WoL isn't enabled (e.g. certain USB network drivers).
>
> The new approach focuses on reducing the risk to break existing stuff.
> We add a flag wol_enabled to struct net_device which is set in
> ethtool_set_wol(). Then this flag is checked in phy_suspend().
> This doesn't cover 100% of the cases yet (e.g. if WoL is enabled w/o
> explicit configuration), but it covers the most relevant cases with
> very little risk of regressions.
>
> v2:
> - Fix a typo
Series applied, thanks.
Please deal with the extra 4 byte size of net_device in net-next.
Thanks.
^ permalink raw reply
* Re: [PATCH net] net/ipv6: Remove extra call to ip6_convert_metrics for multipath case
From: David Miller @ 2018-09-27 2:58 UTC (permalink / raw)
To: dsahern; +Cc: netdev, dsahern
In-Reply-To: <20180927003514.18435-1-dsahern@kernel.org>
From: dsahern@kernel.org
Date: Wed, 26 Sep 2018 17:35:14 -0700
> From: David Ahern <dsahern@gmail.com>
>
> The change to move metrics from the dst to rt6_info moved the call
> to ip6_convert_metrics from ip6_route_add to ip6_route_info_create. In
> doing so it makes the call in ip6_route_info_append redundant and
> actually leaks the metrics installed as part of the ip6_route_info_create.
> Remove the now unnecessary call.
>
> Fixes: d4ead6b34b67f ("net/ipv6: move metrics from dst to rt6_info")
> Signed-off-by: David Ahern <dsahern@gmail.com>
Applied and queued up for -stable, thanks David.
^ permalink raw reply
* Re: Regression caused by commit 7bb05b85bc2d ("r8169: don't use MSI-X on RTL8106e")
From: Jian-Hong Pan @ 2018-09-27 9:03 UTC (permalink / raw)
To: andy.shevchenko
Cc: Kai-Heng Feng, Heiner Kallweit, Thomas Gleixner,
Linux Netdev List, Linux Kernel, Linux Upstreaming Team,
Daniel Drake, Steve Dodd
In-Reply-To: <CAHp75VcCLvb+LvE49YNWnptTMFpUNuj+Jao7udhQeLfOuHh2vg@mail.gmail.com>
Andy Shevchenko <andy.shevchenko@gmail.com> 於 2018年9月22日 週六 上午1:08寫道：
>
> On Thu, Sep 13, 2018 at 8:53 AM Jian-Hong Pan <jian-hong@endlessm.com> wrote:
> >
> > 2018-09-12 16:19 GMT+08:00 Kai-Heng Feng <kai.heng.feng@canonical.com>:
> > > at 14:32, Thomas Gleixner <tglx@linutronix.de> wrote:
> > >
> > >> On Wed, 12 Sep 2018, Kai-Heng Feng wrote:
> > >>
> > >>> There's a Dell machine with RTL8106e stops to work after S3 since the
> > >>> commit introduced. So I am wondering if it's possible to revert the
> > >>> commit and use DMI/subsystem id based quirk table?
> > >>
> > >>
> > >> Probably.
>
> Have you seen this thread:
> https://patchwork.ozlabs.org/cover/968924/
>
> and this one:
> https://patchwork.kernel.org/patch/10583229/
Ya! It is the one. And it is discussed in bugzilla
https://bugzilla.kernel.org/show_bug.cgi?id=201181
Now, the revert patch is submitted https://lkml.org/lkml/2018/9/27/224
However, still thanks for your information. :)
Regards,
Jian-Hong Pan
> ?
>
> > >
> > >
> > > Hopefully Jian-Hong can cook up a quirk table for the issue.
> >
> > Module r8169 gets nothing in the PCI BAR after system resumes which
> > makes MSI-X fail on some ASUS laptops equipped with RTL8106e chip.
> > https://www.spinics.net/lists/linux-pci/msg75598.html
> >
> > Actually, I am waiting for the patch "PCI: Reprogram bridge prefetch
> > registers on resume" being merged.
> > https://marc.info/?l=linux-pm&m=153680987814299&w=2
> >
> > It resolves the drivers which get nothing in PCI BAR after system resumes.
> >
> > After that, I can remove the falling back code of RTL8106e.
> >
> > Heiner, any comment?
> >
> > Regards,
> > Jian-Hong Pan
> >
> > >>
> > >>> It's because of commit bc976233a872 ("genirq/msi, x86/vector: Prevent
> > >>> reservation mode for non maskable MSI") cleared the reservation mode, and
> > >>> I
> > >>> can see this after S3:
> > >>>
> > >>> [ 94.872838] do_IRQ: 3.33 No irq handler for vector
> > >>
> > >>
> > >> It's not because of that commit, really. There is a interrupt sent after
> > >> resume to the wrong vector for whatever reason. The MSI vector cannot be
> > >> masked it seems in the device, but the driver should quiescen the device
> > >> to
> > >> a point where it does not send interrupts.
> > >
> > >
> > > Understood.
> > >
> > >>
> > >>> If the device uses MSI-X instead of MSI, the issue doesn't happen because
> > >>> of
> > >>> reservation mode.
> > >>
> > >>
> > >> Reservation mode has absolutely nothing to do with that. What prevents the
> > >> issue is the fact that MSI-X can be masked by the IRQ core.
> > >
> > >
> > > So in this case I think keep the device using MSI-X is a better route, it's
> > > MSI-X capable anyway.
> > >
> > >>
> > >>> Is it something should be handled by x86 BIOS? Because I don't see this
> > >>> issue
> > >>> when I use Suspend-to-Idle, which doesn't use BIOS to do suspend.
> > >>
> > >>
> > >> Suspend to idle works completely different and I don't see the BIOS at
> > >> fault here. it's more an issue of MSI not being maskable on that device,
> > >> which can't be fixed in BIOS or it's some half quiescened state which is
> > >> used when suspending and that's a pure driver issue.
> > >
> > >
> > > Understood.
> > > Thanks for all the info!
> > >
> > > Kai-Heng
> > >
> > >>
> > >> Thanks,
> > >>
> > >> tglx
> > >
> > >
> > >
>
>
>
> --
> With Best Regards,
> Andy Shevchenko
^ permalink raw reply
* Re: [PATCH net-next] can: xilinx: fix return type of ndo_start_xmit function
From: Marc Kleine-Budde @ 2018-09-27 8:58 UTC (permalink / raw)
To: YueHaibing, davem, wg, michal.simek
Cc: linux-kernel, netdev, linux-can, linux-arm-kernel
In-Reply-To: <20180926103229.13204-1-yuehaibing@huawei.com>
[-- Attachment #1.1: Type: text/plain, Size: 694 bytes --]
On 09/26/2018 12:32 PM, YueHaibing wrote:
> The method ndo_start_xmit() is defined as returning an 'netdev_tx_t',
> which is a typedef for an enum type, so make sure the implementation in
> this driver has returns 'netdev_tx_t' value, and change the function
> return type to netdev_tx_t.
>
> Found by coccinelle.
>
> Signed-off-by: YueHaibing <yuehaibing@huawei.com>
applied to can-next.
Tnx,
Marc
--
Pengutronix e.K. | Marc Kleine-Budde |
Industrial Linux Solutions | Phone: +49-231-2826-924 |
Vertretung West/Dortmund | Fax: +49-5121-206917-5555 |
Amtsgericht Hildesheim, HRA 2686 | http://www.pengutronix.de |
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* Re: [PATCH] hv_netvsc: Make sure out channel is fully opened on send
From: Mohammed Gamal @ 2018-09-27 8:57 UTC (permalink / raw)
To: Haiyang Zhang, Stephen Hemminger, netdev@vger.kernel.org
Cc: otubo@redhat.com, linux-kernel@vger.kernel.org,
devel@linuxdriverproject.org, vkuznets
In-Reply-To: <BN6PR21MB01612C60B5421AF2BB09742CCA150@BN6PR21MB0161.namprd21.prod.outlook.com>
On Wed, 2018-09-26 at 17:13 +0000, Haiyang Zhang wrote:
> > -----Original Message-----
> > From: Mohammed Gamal <mgamal@redhat.com>
> > Sent: Wednesday, September 26, 2018 12:34 PM
> > To: Stephen Hemminger <sthemmin@microsoft.com>; netdev@vger.kernel.
> > org
> > Cc: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> > <haiyangz@microsoft.com>; vkuznets <vkuznets@redhat.com>;
> > otubo@redhat.com; cavery <cavery@redhat.com>; linux-
> > kernel@vger.kernel.org; devel@linuxdriverproject.org; Mohammed
> > Gamal
> > <mgamal@redhat.com>
> > Subject: [PATCH] hv_netvsc: Make sure out channel is fully opened
> > on send
> >
> > Dring high network traffic changes to network interface parameters
> > such as
> > number of channels or MTU can cause a kernel panic with a NULL
> > pointer
> > dereference. This is due to netvsc_device_remove() being called and
> > deallocating the channel ring buffers, which can then be accessed
> > by
> > netvsc_send_pkt() before they're allocated on calling
> > netvsc_device_add()
> >
> > The patch fixes this problem by checking the channel state and
> > returning
> > ENODEV if not yet opened. We also move the call to
> > hv_ringbuf_avail_percent()
> > which may access the uninitialized ring buffer.
> >
> > Signed-off-by: Mohammed Gamal <mgamal@redhat.com>
> > ---
> > drivers/net/hyperv/netvsc.c | 7 ++++++-
> > 1 file changed, 6 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/net/hyperv/netvsc.c
> > b/drivers/net/hyperv/netvsc.c index
> > fe01e14..75f1b31 100644
> > --- a/drivers/net/hyperv/netvsc.c
> > +++ b/drivers/net/hyperv/netvsc.c
> > @@ -825,7 +825,12 @@ static inline int netvsc_send_pkt(
> > struct netdev_queue *txq = netdev_get_tx_queue(ndev,
> > packet->q_idx);
> > u64 req_id;
> > int ret;
> > - u32 ring_avail =
> > hv_get_avail_to_write_percent(&out_channel-
> > > outbound);
> >
> > + u32 ring_avail;
> > +
> > + if (out_channel->state != CHANNEL_OPENED_STATE)
> > + return -ENODEV;
> > +
> > + ring_avail = hv_get_avail_to_write_percent(&out_channel-
> > >outbound);
>
> When you reproducing the NULL ptr panic, does your kernel include the
> following patch?
> hv_netvsc: common detach logic
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/c
> ommit/?id=7b2ee50c0cd513a176a26a71f2989facdd75bfea
>
Yes it is included. And the commit did reduce the occurrence of this
race condition, but it still nevertheless occurs albeit rarely.
> We call netif_tx_disable(ndev) and netif_device_detach(ndev) before
> doing the changes
> on MTU or #channels. So there should be no call to start_xmit() when
> channel is not ready.
>
> If you see the check for CHANNEL_OPENED_STATE is still necessary on
> upstream kernel (including
> the patch " common detach logic "), we should debug further on the
> code and find out the
> root cause.
>
> Thanks,
> - Haiyang
>
_______________________________________________
devel mailing list
devel@linuxdriverproject.org
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
^ permalink raw reply
* Re: [PATCH] netlink: add policy attribute range validation
From: Michal Kubecek @ 2018-09-27 8:48 UTC (permalink / raw)
To: Johannes Berg
Cc: linux-wireless-u79uwXL29TY76Z2rM5mHXA,
netdev-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1538035929.14416.21.camel-cdvu00un1VgdHxzADdlk8Q@public.gmane.org>
On Thu, Sep 27, 2018 at 10:12:09AM +0200, Johannes Berg wrote:
> On Thu, 2018-09-27 at 09:16 +0200, Michal Kubecek wrote:
>
> > The overloading still feels a bit complicated. Perhaps we could rather
> > use validation_data in the natural way, i.e. as a pointer to validation
> > data. That would be a struct (maybe array) of two values of the
> > corresponding type. It would mean a bit more data and a bit more writing
> > but it would be IMHO more straightforward.
>
> I considered that, but I didn't really like it either. The memory
> wasting isn't *that* bad (even if we go to s64 that'd only be ~20x16
> bytes for nl80211, eating up 320 out of the 550 saved, but still); I'm
> more worried about making this really hard to actually *do*.
>
> Consider
>
> policy[] = {
> ...
> [NL80211_ATTR_WIPHY_RETRY_SHORT] =
> NLA_POLICY_RANGE(NLA_U8, 1, 255),
> ...
> };
>
> vs.
>
> static const struct netlink_policy_range retry_range = {
> .min = 1,
> .max = 255,
> };
We could still use helper macros so this part could become
DEFINE_NLA_U8_RANGE(retry_range, 1, 255);
or
DEFINE_NLA_RANGE(retry_range, u8, 1, 255);
>
> policy[] = {
> ...
> [NL80211_ATTR_WIPHY_RETRY_SHORT] = {
> .type = NLA_U8,
> .validation_data = &retry_range,
> },
> ...
> };
And this could be also shortened using a macro.
It would still be longer but not that much.
> That's significantly more to type, to the point where I'd seriously
> consider doing this only for attributes that are used and checked in
> many places - it doesn't feel like a big win over manual range-checking.
>
> But I want it to be a win over manual range-checking so it gets used
> more because it's more efficient, less prone to getting messed up if
> multiple places use the same attribute and validates attributes even if
> they're ignored by an operation.
>
>
> I'd also say that we're certainly no strangers to union/overloading, so
> I don't feel like this is a big argument. One doesn't even really have
> to be *aware* of it for the most part: if it were a struct instead of a
> union, it'd actually have the same effect since the .type field
> indicates which part gets used. That it's overloaded in a union is
> basically just a space saving measure, I don't think it makes the
> reasoning much more complex?
I didn't mean it as a serious objection, rather a note that the gain may
not be worth the additional complexity. But if you want to follow in the
direction you indicated later (in particular, allowing different
interpretations of validation_data for the same type), overloading does
indeed make more sense.
Michal Kubecek
^ permalink raw reply
* [PATCH net] vhost-vsock: fix use after free
From: Jason Wang @ 2018-09-27 8:43 UTC (permalink / raw)
To: stefanha, mst; +Cc: kvm, virtualization, netdev, linux-kernel, Jason Wang
The access of vsock is not protected by vhost_vsock_lock. This may
lead use after free since vhost_vsock_dev_release() may free the
pointer at the same time.
Fix this by holding the lock during the acess.
Reported-by: syzbot+e3e074963495f92a89ed@syzkaller.appspotmail.com
Fixes: 16320f363ae1 ("vhost-vsock: add pkt cancel capability")
Fixes: 433fc58e6bf2 ("VSOCK: Introduce vhost_vsock.ko")
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
- The patch is needed for -stable.
---
drivers/vhost/vsock.c | 26 +++++++++++++++++++-------
1 file changed, 19 insertions(+), 7 deletions(-)
diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
index 34bc3ab40c6d..7d0b292867fd 100644
--- a/drivers/vhost/vsock.c
+++ b/drivers/vhost/vsock.c
@@ -210,21 +210,27 @@ vhost_transport_send_pkt(struct virtio_vsock_pkt *pkt)
struct vhost_vsock *vsock;
int len = pkt->len;
+ spin_lock_bh(&vhost_vsock_lock);
+
/* Find the vhost_vsock according to guest context id */
- vsock = vhost_vsock_get(le64_to_cpu(pkt->hdr.dst_cid));
+ vsock = __vhost_vsock_get(le64_to_cpu(pkt->hdr.dst_cid));
if (!vsock) {
virtio_transport_free_pkt(pkt);
+ spin_unlock_bh(&vhost_vsock_lock);
return -ENODEV;
}
if (pkt->reply)
atomic_inc(&vsock->queued_replies);
- spin_lock_bh(&vsock->send_pkt_list_lock);
+ spin_lock(&vsock->send_pkt_list_lock);
list_add_tail(&pkt->list, &vsock->send_pkt_list);
- spin_unlock_bh(&vsock->send_pkt_list_lock);
+ spin_unlock(&vsock->send_pkt_list_lock);
vhost_work_queue(&vsock->dev, &vsock->send_pkt_work);
+
+ spin_unlock_bh(&vhost_vsock_lock);
+
return len;
}
@@ -236,18 +242,22 @@ vhost_transport_cancel_pkt(struct vsock_sock *vsk)
int cnt = 0;
LIST_HEAD(freeme);
+ spin_lock_bh(&vhost_vsock_lock);
+
/* Find the vhost_vsock according to guest context id */
- vsock = vhost_vsock_get(vsk->remote_addr.svm_cid);
- if (!vsock)
+ vsock = __vhost_vsock_get(vsk->remote_addr.svm_cid);
+ if (!vsock) {
+ spin_unlock_bh(&vhost_vsock_lock);
return -ENODEV;
+ }
- spin_lock_bh(&vsock->send_pkt_list_lock);
+ spin_lock(&vsock->send_pkt_list_lock);
list_for_each_entry_safe(pkt, n, &vsock->send_pkt_list, list) {
if (pkt->vsk != vsk)
continue;
list_move(&pkt->list, &freeme);
}
- spin_unlock_bh(&vsock->send_pkt_list_lock);
+ spin_unlock(&vsock->send_pkt_list_lock);
list_for_each_entry_safe(pkt, n, &freeme, list) {
if (pkt->reply)
@@ -265,6 +275,8 @@ vhost_transport_cancel_pkt(struct vsock_sock *vsk)
vhost_poll_queue(&tx_vq->poll);
}
+ spin_unlock_bh(&vhost_vsock_lock);
+
return 0;
}
--
2.17.1
^ permalink raw reply related
* Could your company tell us if you can supply us
From: DEBENHAMS PLC @ 2018-09-27 0:05 UTC (permalink / raw)
Hi
Situated in (United Kingdom) DEBENHAMS RETAIL PLC is a retailer shop
furnished by European products.We are looking to buy ( Fashion
items,Perfumes, Lighters, Tiles Marbles, Wooden Floors, Smart-phones,
Tablets,Laptops, TVs, Air conditioner etc...) and find new partnership
with companies dealing with different products so can you please send us
your Catalog or your website to learn more about your products or prices
list by email and if we can make some order with you and start long-term
partnership.Could you also tell us if you can supply us, more
information about the possibility to become one of your regular
customers?
Our Payment Policy is payment by swift within 30 days net in Europe.
Waiting for your feedback.
Best regards.
Richard Jones
Trading Director, Global Sourcing
DEBENHAMS RETAIL PLC
Direct line : +44 7413339961
Fax : +44 02082534105
www.debenhams.com
www.debenhamsplus.com
^ permalink raw reply
* Re: [PATCH bpf-next 0/5] Introduce libbpf_attach_type_by_name
From: Jakub Kicinski @ 2018-09-27 2:06 UTC (permalink / raw)
To: Andrey Ignatov
Cc: netdev@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net,
Kernel Team
In-Reply-To: <20180926235413.GA38328@rdna-mbp.dhcp.thefacebook.com>
On Wed, 26 Sep 2018 23:54:17 +0000, Andrey Ignatov wrote:
> Jakub Kicinski <jakub.kicinski@netronome.com> [Wed, 2018-09-26 16:20 -0700]:
> > On Wed, 26 Sep 2018 15:24:52 -0700, Andrey Ignatov wrote:
> > > This patch set introduces libbpf_attach_type_by_name function in libbpf to
> > > identify attach type by section name.
> > >
> > > This is useful to avoid writing same logic over and over again in user
> > > space applications that leverage libbpf.
> > >
> > > Patch 1 has more details on the new function and problem being solved.
> > > Patches 2 and 3 add support for new section names.
> > > Patch 4 uses new function in a selftest.
> > > Patch 5 adds selftest for libbpf_{prog,attach}_type_by_name.
> > >
> > > As a side note there are a lot of inconsistencies now between names used by
> > > libbpf and bpftool (e.g. cgroup/skb vs cgroup_skb, cgroup_device and device
> > > vs cgroup/dev, sockops vs sock_ops, etc). This patch set does not address
> > > it but it tries not to make it harder to address it in the future.
> >
> > I was wondering a few times whether I should point it out to people
> > during review, but thought it would be nit picking. Maybe we should be
> > more strict.
> >
> > Your series LGTM!
>
> Thanks for review!
>
> IMO having it consistent would be great, e.g. one writes a program with
> section name X and bpftool shows/accepts it in exactly same way in all
> its sub-commands (w/o maybe custom suffix added by program writer).
>
> But I doubt that keeping a few places in sync manually will work long
> term since it's easy to miss such a thing.
>
> What do you think of having one source of truth in libbpf so that a
> string for prog_type or attach_type is defined once and all other places
> (e.g. bpftool prog show, bpftool cgroup show) use only corresponding
> enum-s to get those strings, but don't introduce any new strings?
>
> Keeping already existing names in a backward compatible way is a pain
> though.
One source of truth would be nice IMO. The backward compat ties our
hands a tiny bit, but we could do a fallback strategy, where bpftool
checks if it has a name defined for a type, and if it doesn't (for new
types) it will ask libbpf.
There is another place actually where names are hard coded - for program
load command we use the types as they appear in libbpf already. And
that requires putting them in help, man page and bash completions, too.
> Another thing, I was wondering, is if there is a way to bypass strings
> completely (at least in libbpf, since bpftool still has to print
> human-readable names) and keep actual bpf_prog_type and bpf_attach_type
> as metadata for a program in ELF file. Maybe some compiler magic ..
Possibly. I'm far from an ELF expert, but I think attaching arbitrary
metadata to sections may be harder, name is already there and is free
form.
^ permalink raw reply
* Re: [PATCH] netlink: add policy attribute range validation
From: Johannes Berg @ 2018-09-27 8:12 UTC (permalink / raw)
To: Michal Kubecek
Cc: linux-wireless-u79uwXL29TY76Z2rM5mHXA,
netdev-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20180927071621.GF30601-OEaqT8BN2ewCVLCxKZUutA@public.gmane.org>
On Thu, 2018-09-27 at 09:16 +0200, Michal Kubecek wrote:
> The overloading still feels a bit complicated. Perhaps we could rather
> use validation_data in the natural way, i.e. as a pointer to validation
> data. That would be a struct (maybe array) of two values of the
> corresponding type. It would mean a bit more data and a bit more writing
> but it would be IMHO more straightforward.
I considered that, but I didn't really like it either. The memory
wasting isn't *that* bad (even if we go to s64 that'd only be ~20x16
bytes for nl80211, eating up 320 out of the 550 saved, but still); I'm
more worried about making this really hard to actually *do*.
Consider
policy[] = {
...
[NL80211_ATTR_WIPHY_RETRY_SHORT] =
NLA_POLICY_RANGE(NLA_U8, 1, 255),
...
};
vs.
static const struct netlink_policy_range retry_range = {
.min = 1,
.max = 255,
};
policy[] = {
...
[NL80211_ATTR_WIPHY_RETRY_SHORT] = {
.type = NLA_U8,
.validation_data = &retry_range,
},
...
};
That's significantly more to type, to the point where I'd seriously
consider doing this only for attributes that are used and checked in
many places - it doesn't feel like a big win over manual range-checking.
But I want it to be a win over manual range-checking so it gets used
more because it's more efficient, less prone to getting messed up if
multiple places use the same attribute and validates attributes even if
they're ignored by an operation.
I'd also say that we're certainly no strangers to union/overloading, so
I don't feel like this is a big argument. One doesn't even really have
to be *aware* of it for the most part: if it were a struct instead of a
union, it'd actually have the same effect since the .type field
indicates which part gets used. That it's overloaded in a union is
basically just a space saving measure, I don't think it makes the
reasoning much more complex?
That said, given that I also later sent that RFC patch for a further
validation function pointer (which is useful e.g. for ensuring certain
binary attributes are well-formed), I think it might in fact be better
to split .type field (it really only needs to be u8, we have ~20) and
adding a "validation" enum to the policy:
enum netlink_policy_validation {
/* default */
NLA_VALIDATE_NONE,
/* valid for NLA_{U,S}{8,16,32,64} */
NLA_VALIDATE_MIN,
NLA_VALIDATE_MAX,
NLA_VALIDATE_RANGE,
/* valid for any type other than NLA_BITFIELD32/NLA_REJECT */
NLA_VALIDATE_FUNCTION,
};
Combining that with macros like the ones I wrote in my previous message
in this thread:
#define __NLA_ENSURE(condition) (sizeof(char[1 - 2*!(condition)]) - 1)
#define NLA_ENSURE_INT_TYPE(tp) \
(__NLA_ENSURE(tp == NLA_S8 || tp == NLA_U8 || \
tp == NLA_S16 || tp == NLA_U16 || \
tp == NLA_S32 || tp == NLA_U32 || \
tp == NLA_S64 || tp == NLA_U64) + tp)
#define NLA_ENSURE_NO_VALIDATION_PTR(tp) \
(__NLA_ENSURE(tp != NLA_BITFIELD32 && \
tp != NLA_REJECT && \
tp != NLA_NESTED && \
tp != NLA_NESTED_ARRAY) + tp)
#define NLA_POLICY_RANGE(tp, _min, _max) {
.type = NLA_ENSURE_INT_TYPE(tp),
.validate_type = NLA_VALIDATE_RANGE,
.min = _min,
.max = _max,
}
#define NLA_POLICY_MIN(tp, _min) {
.type = NLA_ENSURE_INT_TYPE(tp),
.validate_type = NLA_VALIDATE_MIN,
.min = _min,
}
#define NLA_POLICY_MAX(tp, _max) {
.type = NLA_ENSURE_INT_TYPE(tp),
.validate_type = NLA_VALIDATE_MAX,
.max = _max,
}
#define NLA_POLICY_FN(tp, fn) {
.type = NLA_ENSURE_NO_VALIDATION_PTR(tp),
.validate_type = NLA_VALIDATE_FUNCTION,
.validate = fn,
}
This would even give us the flexibility to extend the validation type
further in the future, to actually have what you suggested where the
validation_data is a pointer and the valid range is given in a struct
pointed to, to allow larger ranges than s16.
johannes
^ permalink raw reply
* Re: KASAN: use-after-free Read in tcf_block_find
From: Dmitry Vyukov @ 2018-09-27 8:10 UTC (permalink / raw)
To: Cong Wang
Cc: Eric Dumazet, syzbot+37b8770e6d5a8220a039, David Miller,
Jamal Hadi Salim, Jiri Pirko, LKML,
Linux Kernel Network Developers, syzkaller-bugs
In-Reply-To: <CACT4Y+YbfOYDHJjeK-88wQDg2oSF3fqsaWBhkDwnDr+KUsLnDA@mail.gmail.com>
On Thu, Sep 27, 2018 at 10:06 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Wed, Sep 26, 2018 at 11:55 PM, Cong Wang <xiyou.wangcong@gmail.com> wrote:
>>> >> Hello,
>>> >>
>>> >> syzbot found the following crash on:
>>> >>
>>> >> HEAD commit: 4b1bd6976945 net: phy: marvell: Fix build.
>>> >> git tree: net-next
>>> >> console output: https://syzkaller.appspot.com/x/log.txt?x=16f763fa400000
>>> >> kernel config: https://syzkaller.appspot.com/x/.config?x=443816db871edd66
>>> >> dashboard link: https://syzkaller.appspot.com/bug?extid=37b8770e6d5a8220a039
>>> >> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
>>> >> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=17a5614e400000
>>> >> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=141a532a400000
>>> >>
>>> >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>>> >> Reported-by: syzbot+37b8770e6d5a8220a039@syzkaller.appspotmail.com
>>> >>
>>> >> IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
>>> >> 8021q: adding VLAN 0 to HW filter on device team0
>>> >> ==================================================================
>>> >> BUG: KASAN: use-after-free in tcf_block_find+0x9d1/0xb90
>>> >> net/sched/cls_api.c:646
>>> >> Read of size 4 at addr ffff8801cc126978 by task syz-executor002/5646
>>> >>
>>> >> CPU: 1 PID: 5646 Comm: syz-executor002 Not tainted 4.19.0-rc5+ #232
>>> >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>>> >> Google 01/01/2011
>>> >> Call Trace:
>>> >> __dump_stack lib/dump_stack.c:77 [inline]
>>> >> dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
>>> >> print_address_description.cold.8+0x9/0x1ff mm/kasan/report.c:256
>>> >> kasan_report_error mm/kasan/report.c:354 [inline]
>>> >> kasan_report.cold.9+0x242/0x309 mm/kasan/report.c:412
>>> >> __asan_report_load4_noabort+0x14/0x20 mm/kasan/report.c:432
>>> >> tcf_block_find+0x9d1/0xb90 net/sched/cls_api.c:646
>>> >
>>> > Hmm. looks like missing a rcu_dereference() here:
>>> >
>>> > if (!*parent) {
>>> > *q = dev->qdisc;
>>> > *parent = (*q)->handle;
>>> >
>>>
>>> We hold RTNL here.
>>>
>>> Have we already done the changes allowing dev->qdisc being changed
>>> without RTNL being required ?
>>
>> That transition is not completed yet, but holding RTNL doesn't help
>> any more after we now free qdisc in rcu callback.
>>
>> More interestingly, rcu read lock is already held in this context,
>> so it seems we still have to use rcu_deference() to read dev->qdisc
>> and of course mark it with __rcu too.
>
> On hardware level rcu_deference() only affects Alpha. On software
> level, it can affect all archs due to e.g. a re-read of the variable.
> Do we see the potential for compilation that would lead to the UAF
> here? A missed rcu_deference() would not be the first thing that I
> would suspect for UAF on x86. It's more likely some bolder bug.
Would a stack trace for call_rcu be helpful here? I have this idea for
a long time, but never get around to implementing it:
https://bugzilla.kernel.org/show_bug.cgi?id=198437
Also FWIW I recently used the following hack for another net bug. It
made that other bug involving call_rcu way more likely to fire. Maybe
it will be helpful here too.
diff --git a/net/core/dst.c b/net/core/dst.c
index 81ccf20e28265..591a8d0aca545 100644
--- a/net/core/dst.c
+++ b/net/core/dst.c
@@ -187,8 +187,16 @@ void dst_release(struct dst_entry *dst)
if (unlikely(newrefcnt < 0))
net_warn_ratelimited("%s: dst:%p refcnt:%d\n",
__func__, dst, newrefcnt);
- if (!newrefcnt)
- call_rcu(&dst->rcu_head, dst_destroy_rcu);
+ if (!newrefcnt) {
+ if (lock_is_held(&rcu_bh_lock_map) ||
+ lock_is_held(&rcu_lock_map) ||
+ lock_is_held(&rcu_sched_lock_map)) {
+ call_rcu(&dst->rcu_head, dst_destroy_rcu);
+ } else {
+ synchronize_rcu();
+ dst_destroy_rcu(&dst->rcu_head);
+ }
+ }
}
}
^ permalink raw reply related
* Re: KASAN: use-after-free Read in tcf_block_find
From: Dmitry Vyukov @ 2018-09-27 8:06 UTC (permalink / raw)
To: Cong Wang
Cc: Eric Dumazet, syzbot+37b8770e6d5a8220a039, David Miller,
Jamal Hadi Salim, Jiri Pirko, LKML,
Linux Kernel Network Developers, syzkaller-bugs
In-Reply-To: <CAM_iQpX4sKSPFXgEjU-be=he=91+=3Bm9KvP3CMciA9zi2b-6g@mail.gmail.com>
On Wed, Sep 26, 2018 at 11:55 PM, Cong Wang <xiyou.wangcong@gmail.com> wrote:
>> >> Hello,
>> >>
>> >> syzbot found the following crash on:
>> >>
>> >> HEAD commit: 4b1bd6976945 net: phy: marvell: Fix build.
>> >> git tree: net-next
>> >> console output: https://syzkaller.appspot.com/x/log.txt?x=16f763fa400000
>> >> kernel config: https://syzkaller.appspot.com/x/.config?x=443816db871edd66
>> >> dashboard link: https://syzkaller.appspot.com/bug?extid=37b8770e6d5a8220a039
>> >> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
>> >> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=17a5614e400000
>> >> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=141a532a400000
>> >>
>> >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>> >> Reported-by: syzbot+37b8770e6d5a8220a039@syzkaller.appspotmail.com
>> >>
>> >> IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
>> >> 8021q: adding VLAN 0 to HW filter on device team0
>> >> ==================================================================
>> >> BUG: KASAN: use-after-free in tcf_block_find+0x9d1/0xb90
>> >> net/sched/cls_api.c:646
>> >> Read of size 4 at addr ffff8801cc126978 by task syz-executor002/5646
>> >>
>> >> CPU: 1 PID: 5646 Comm: syz-executor002 Not tainted 4.19.0-rc5+ #232
>> >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>> >> Google 01/01/2011
>> >> Call Trace:
>> >> __dump_stack lib/dump_stack.c:77 [inline]
>> >> dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
>> >> print_address_description.cold.8+0x9/0x1ff mm/kasan/report.c:256
>> >> kasan_report_error mm/kasan/report.c:354 [inline]
>> >> kasan_report.cold.9+0x242/0x309 mm/kasan/report.c:412
>> >> __asan_report_load4_noabort+0x14/0x20 mm/kasan/report.c:432
>> >> tcf_block_find+0x9d1/0xb90 net/sched/cls_api.c:646
>> >
>> > Hmm. looks like missing a rcu_dereference() here:
>> >
>> > if (!*parent) {
>> > *q = dev->qdisc;
>> > *parent = (*q)->handle;
>> >
>>
>> We hold RTNL here.
>>
>> Have we already done the changes allowing dev->qdisc being changed
>> without RTNL being required ?
>
> That transition is not completed yet, but holding RTNL doesn't help
> any more after we now free qdisc in rcu callback.
>
> More interestingly, rcu read lock is already held in this context,
> so it seems we still have to use rcu_deference() to read dev->qdisc
> and of course mark it with __rcu too.
On hardware level rcu_deference() only affects Alpha. On software
level, it can affect all archs due to e.g. a re-read of the variable.
Do we see the potential for compilation that would lead to the UAF
here? A missed rcu_deference() would not be the first thing that I
would suspect for UAF on x86. It's more likely some bolder bug.
^ permalink raw reply
* nfs+tcp boot failures on linuxnext-20180924 from switch tcp_clock_ns to CLOCK_TAI
From: Leonard Crestez @ 2018-09-27 1:15 UTC (permalink / raw)
To: sfr@canb.auug.org.au, davem@davemloft.net, edumazet@google.com
Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-next@vger.kernel.org
Hello,
It seems that after commit 72b0094f9182 ("tcp: switch tcp_clock_ns() to
CLOCK_TAI base") some systems that use nfs over tcp fail to boot. The
last line printed in the log is from systemd:
[ 7.232579] systemd[1]: System time before build time, advancing clock.
Superficially it looks like very large clock discontinuities now break
TCP. Maybe boottime could avoid such issues?
I didn't find similar reports anywhere else. The machines I’m seeing
this are all 32bit arm imx (this shouldn’t matter) and it seems their
RTC isn’t properly setup so they boot with realtime set to unix zero,
then a ~50 years jump happens when systemd starts up. This is the
likely trigger for this issue.
--
Regards,
Leonard
^ permalink raw reply
* Reminder,
From: Juliet Muhammad @ 2018-09-27 0:50 UTC (permalink / raw)
To: Recipients
Hello
Please i still await your response regarding my previous email.
^ permalink raw reply
* Reminder,
From: Juliet Muhammad @ 2018-09-27 0:45 UTC (permalink / raw)
To: Recipients
Hello
Please i still await your response regarding my previous email.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox