* Re: SQ overflow seen running isert traffic
From: Jason Gunthorpe @ 2016-10-18 16:13 UTC (permalink / raw)
To: Steve Wise
Cc: 'Sagi Grimberg', 'Potnuri Bharat Teja',
target-devel, nab, linux-rdma
In-Reply-To: <005101d2294c$be5bb460$3b131d20$@opengridcomputing.com>
On Tue, Oct 18, 2016 at 09:34:33AM -0500, Steve Wise wrote:
> 1) we believe the iSER + RW API correctly sizes the SQ, yet we're seeing SQ
> overflows. So the SQ sizing needs more investigation.
NFS had this sort of problem - in that case it was because the code
was assuming that a RQ completion implied SQ space - that is not
legal, only direct completions from SQ WCs can guide available space
in the SQ..
Jason
^ permalink raw reply
* Re: building rdma-core from travis
From: Jason Gunthorpe @ 2016-10-18 16:07 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Hefty, Sean,
linux-rdma (linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org)
In-Reply-To: <20161018050834.GF25013-2ukJVAZIZ/Y@public.gmane.org>
On Tue, Oct 18, 2016 at 08:08:34AM +0300, Leon Romanovsky wrote:
> > > I can see that travis runs successfully against rdma-core. Was there a
> > > magic setting used to work-around the travis version?
> >
> > It looks like rdma-core requires the 'trusty' build on travis...
>
> Yes, you are right.
> "# We need at least cmake 2.12, this means we need to use trusty."
> http://marc.info/?l=linux-rdma&m=147587977913428&w=2
Right. rdma-core does not support precise, at all. Aside from cmake,
the original source packages had some non-functional 'compat' for such
an old glibc (from 2012!) that I stripped out as well.
I can't recommend that anybody working on a c-language project should
be using the precise images on travis for anything <shrug>
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v3 net-next 00/11] net: Fix netdev adjacency tracking
From: David Ahern @ 2016-10-18 15:58 UTC (permalink / raw)
To: David Miller
Cc: jiri, netdev, dledford, sean.hefty, hal.rosenstock, linux-rdma,
j.vosburgh, vfalico, andy, jeffrey.t.kirsher, intel-wired-lan
In-Reply-To: <20161018.114631.1954199143257806104.davem@davemloft.net>
On 10/18/16 9:46 AM, David Miller wrote:
> Series applied, but the recursion is disappointing.
>
> If we run into problems due to kernel stack depth because of this with
> some configurations (reasonable or not, if we allow it then it can't
> crash the kernel), we will either need to find a way to make this walk
> iterative or revert these changes.
understood.
Since 4.9 is tagged as the next LTS I would like to see the series applied to it at some point - assuming no problems show up with wider exposure in net-next.
^ permalink raw reply
* Re: ???For help??? configure crossbar build tool in CMakelist.txt
From: Jason Gunthorpe @ 2016-10-18 15:50 UTC (permalink / raw)
To: oulijun; +Cc: linux-rdma, Linuxarm
In-Reply-To: <5805FF8C.1060902-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
On Tue, Oct 18, 2016 at 06:55:08PM +0800, oulijun wrote:
> if I use crossbar build tool aarch64-linux-gnu-gcc for building
> the directory(provider/hns), what i should do it ?
You cannot cross compile only a part of the project, you must cross
compile everything.
$ mkdir build; cd build
$ CC=aarch64-linux-gnu-gcc cmake -GNinja
$ ninja
I haven't extensively tested cross compiling, you may run into
problems, in particular I know pkgconfig for libnl doesn't work
reliably when cross compiling.
You may need to build with -DENABLE_RESOLVE_NEIGH=0 if you don't have
a cross compiled libnl3 available.
> My modification currently according to the others as fllows:
>
> in the file : provider/hns/CMakelist.txt
>
> set(CMAKE_C_COMPILER /opt/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux/bin/aarch64-linux-gnu-gcc)
This is not the right approach, the hns provider CMakelist should
detect the compiler is not ARM64 and just do nothing.
Perhaps something like this:
CHECK_C_SOURCE_COMPILES("
#ifndef __ARM64__
#error Failed
#endif
int main(int argc,const char *argv[]) { return 1; }"
HAVE_ARCH_ARM64)
if (HAVE_ARCH_ARM64)
[..]
endif()
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v3 net-next 00/11] net: Fix netdev adjacency tracking
From: David Miller @ 2016-10-18 15:46 UTC (permalink / raw)
To: dsa-qUQiAmfTcIp+XZJcv9eMoEEOCMrvLtNR
Cc: jiri-VPRAkNaXOzVWk0Htik3J/w, netdev-u79uwXL29TY76Z2rM5mHXA,
dledford-H+wXaHxf7aLQT0dZR+AlfA,
sean.hefty-ral2JQCrhuEAvxtiuMwx3w,
hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
j.vosburgh-Re5JQEeQqe8AvxtiuMwx3w, vfalico-Re5JQEeQqe8AvxtiuMwx3w,
andy-QlMahl40kYEqcZcGjlUOXw,
jeffrey.t.kirsher-ral2JQCrhuEAvxtiuMwx3w,
intel-wired-lan-qjLDD68F18P21nG7glBr7A
In-Reply-To: <1476756953-30923-1-git-send-email-dsa-qUQiAmfTcIp+XZJcv9eMoEEOCMrvLtNR@public.gmane.org>
From: David Ahern <dsa-qUQiAmfTcIp+XZJcv9eMoEEOCMrvLtNR@public.gmane.org>
Date: Mon, 17 Oct 2016 19:15:42 -0700
> The netdev adjacency tracking is failing to create proper dependencies
> for some topologies. For example this topology
...
> hits 1 of 2 problems depending on the order of enslavement. The base set of
> commands for both cases:
...
> Case 1 enslave macvlan to the vrf before enslaving the bond to the bridge:
...
> Attempts to delete the VRF:
> ip link delete myvrf
>
> trigger the BUG in __netdev_adjacent_dev_remove:
...
> When the BUG is converted to a WARN_ON it shows 4 missing adjacencies:
> eth3 - myvrf, mvrf - eth3, bond1 - myvrf and myvrf - bond1
>
> All of those are because the __netdev_upper_dev_link function does not
> properly link macvlan lower devices to myvrf when it is enslaved.
...
> Rather than try to maintain a linked list of all upper and lower devices
> per netdevice, only track the direct neighbors. The remaining stack can
> be determined by recursively walking the neighbors.
>
> The existing netdev_for_each_all_upper_dev_rcu,
> netdev_for_each_all_lower_dev and netdev_for_each_all_lower_dev_rcu macros
> are replaced with APIs that walk the upper and lower device lists. The
> new APIs take a callback function and a data arg that is passed to the
> callback for each device in the list. Drivers using the old macros are
> converted in separate patches to make it easier on reviewers. It is an
> API conversion only; no functional change is intended.
Series applied, but the recursion is disappointing.
If we run into problems due to kernel stack depth because of this with
some configurations (reasonable or not, if we allow it then it can't
crash the kernel), we will either need to find a way to make this walk
iterative or revert these changes.
Thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 00/10] mm: adjust get_user_pages* functions to explicitly pass FOLL_* flags
From: Michal Hocko @ 2016-10-18 15:30 UTC (permalink / raw)
To: Lorenzo Stoakes
Cc: linux-mm, Linus Torvalds, Jan Kara, Hugh Dickins, Dave Hansen,
Rik van Riel, Mel Gorman, Andrew Morton, adi-buildroot-devel,
ceph-devel, dri-devel, intel-gfx, kvm, linux-alpha,
linux-arm-kernel, linux-cris-kernel, linux-fbdev, linux-fsdevel,
linux-ia64, linux-kernel, linux-media, linux-mips, linux-rdma,
linux-s390, linux-samsung-soc, linux-scsi, linux-security-module,
linux-sh@
In-Reply-To: <20161013002020.3062-1-lstoakes@gmail.com>
On Thu 13-10-16 01:20:10, Lorenzo Stoakes wrote:
> This patch series adjusts functions in the get_user_pages* family such that
> desired FOLL_* flags are passed as an argument rather than implied by flags.
>
> The purpose of this change is to make the use of FOLL_FORCE explicit so it is
> easier to grep for and clearer to callers that this flag is being used. The use
> of FOLL_FORCE is an issue as it overrides missing VM_READ/VM_WRITE flags for the
> VMA whose pages we are reading from/writing to, which can result in surprising
> behaviour.
>
> The patch series came out of the discussion around commit 38e0885, which
> addressed a BUG_ON() being triggered when a page was faulted in with PROT_NONE
> set but having been overridden by FOLL_FORCE. do_numa_page() was run on the
> assumption the page _must_ be one marked for NUMA node migration as an actual
> PROT_NONE page would have been dealt with prior to this code path, however
> FOLL_FORCE introduced a situation where this assumption did not hold.
>
> See https://marc.info/?l=linux-mm&m=147585445805166 for the patch proposal.
I like this cleanup. Tracking FOLL_FORCE users was always a nightmare
and the flag behavior is really subtle so we should better be explicit
about it. I haven't gone through each patch separately but rather
applied the whole series and checked the resulting diff. This all seems
OK to me and feel free to add
Acked-by: Michal Hocko <mhocko@suse.com>
I am wondering whether we can go further. E.g. it is not really clear to
me whether we need an explicit FOLL_REMOTE when we can in fact check
mm != current->mm and imply that. Maybe there are some contexts which
wouldn't work, I haven't checked.
Then I am also wondering about FOLL_TOUCH behavior.
__get_user_pages_unlocked has only few callers which used to be
get_user_pages_unlocked before 1e9877902dc7e ("mm/gup: Introduce
get_user_pages_remote()"). To me a dropped FOLL_TOUCH seems
unintentional. Now that get_user_pages_unlocked has gup_flags argument I
guess we might want to get rid of the __g-u-p-u version altogether, no?
__get_user_pages is quite low level and imho shouldn't be exported. It's
only user - kvm - should rather pull those two functions to gup instead
and export them. There is nothing really KVM specific in them.
I also cannot say I would be entirely thrilled about get_user_pages_locked,
we only have one user which can simply do lock g-u-p unlock AFAICS.
I guess there is more work in that area and I do not want to impose all
that work on you, but I couldn't resist once I saw you playing in that
area ;) Definitely a good start!
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH rdma-core 2/4] glue/redhat: add udev/systemd/etc infrastructure bits
From: Jarod Wilson @ 2016-10-18 14:51 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Weiny, Ira, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Doug Ledford, Hefty, Sean
In-Reply-To: <20161017201309.GA24291-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
On Mon, Oct 17, 2016 at 02:13:09PM -0600, Jason Gunthorpe wrote:
> On Mon, Oct 17, 2016 at 07:10:46PM +0000, Weiny, Ira wrote:
>
> > What I have been worried about is conflicts between infiniband-diags
> > and the new rdma-core. RH made a separate package out of rdma-ndd
> > so that would be easy but I don't think other distros have.
>
> > So how do you obsolete "part" of a package?
>
> The distros know how to do this, they just conflict with the old
> version of infiniband-diags.
Yeah, basically, you spin up a new infiniband-diags package that doesn't
have rdma-ndd and a new rdma-core package that includes it, along with (in
rpm-ese) 'Conflicts: infiniband-diags < <first version w/o rdma-ndd>', and
that should pretty much cover it.
--
Jarod Wilson
jarod-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH rdma-core 4/4] glue/redhat/spec: build split rpm packages
From: Jarod Wilson @ 2016-10-18 14:44 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA,
dledford-H+wXaHxf7aLQT0dZR+AlfA
In-Reply-To: <20161017190705.GB8122-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
On Mon, Oct 17, 2016 at 01:07:05PM -0600, Jason Gunthorpe wrote:
> On Mon, Oct 17, 2016 at 02:45:06PM -0400, Jarod Wilson wrote:
>
> > > > Wrote: /home/jwilson/rpmbuild/RPMS/x86_64/libcxgb3-11-1.el7.x86_64.rpm
> > > > Wrote: /home/jwilson/rpmbuild/RPMS/x86_64/libcxgb4-11-1.el7.x86_64.rpm
> > > > Wrote: /home/jwilson/rpmbuild/RPMS/x86_64/libhfi1-11-1.el7.x86_64.rpm
> > > > Wrote: /home/jwilson/rpmbuild/RPMS/x86_64/libi40iw-11-1.el7.x86_64.rpm
> > > > Wrote: /home/jwilson/rpmbuild/RPMS/x86_64/libipathverbs-11-1.el7.x86_64.rpm
> > > > Wrote: /home/jwilson/rpmbuild/RPMS/x86_64/libmlx4-11-1.el7.x86_64.rpm
> > > > Wrote: /home/jwilson/rpmbuild/RPMS/x86_64/libmlx5-11-1.el7.x86_64.rpm
> > > > Wrote: /home/jwilson/rpmbuild/RPMS/x86_64/libmthca-11-1.el7.x86_64.rpm
> > > > Wrote: /home/jwilson/rpmbuild/RPMS/x86_64/libnes-11-1.el7.x86_64.rpm
> > > > Wrote: /home/jwilson/rpmbuild/RPMS/x86_64/libocrdma-11-1.el7.x86_64.rpm
> > > > Wrote: /home/jwilson/rpmbuild/RPMS/x86_64/librxe-11-1.el7.x86_64.rpm
> > >
> > > Does this mean RH is not going to do a single package for the
> > > providers? You know another 5 or so are coming..
> >
> > Putting all the providers in a single sub-package is certainly something
> > we could do. With rpm virtual Provides and Obsoletes, we could make the
> > upgrade path transparent to at least yum/dnf/rpm, but it might still
> > confuse people who go looking for the same package they've always had
> > installed, so it could be something we do in Fedora and the next RHEL
> > major release, rather than in a current RHEL minor update.
>
> I would strongly encoruage this - the current scheme is silly, you
> need to know to look for the right provider to get things working
> (which is very much unlike the way the kernel modules work).
>
> Suggest to dump the providers in libiverbs or add a ibverbs-providers
> package..
I'm playing with a setup where libibverbs includes all the providers now.
I think I like it. It'd require some documentation to explain where libfoo
went, but it doesn't look too bad:
$ sudo rpm -Uvh libibverbs-11-1.el7.x86_64.rpm libibverbs-utils-11-1.el7.x86_64.rpm
Preparing... ################################# [100%]
Updating / installing...
1:libibverbs-11-1.el7 ################################# [ 14%]
2:libibverbs-utils-11-1.el7 ################################# [ 29%]
Cleaning up / removing...
3:libibverbs-utils-1.2.1-1.el7 ################################# [ 43%]
4:libi40iw-0.5.227-2.el7 ################################# [ 57%]
5:libocrdma-1.0.8-1.el7 ################################# [ 71%]
6:libmlx5-1.2.1-8.el7 ################################# [ 86%]
7:libibverbs-1.2.1-1.el7 ################################# [100%]
Seems I have the Provides/Obsoletes correct at least. This also eliminates
the somewhat gross libibverbs-driver Provides from each driver and the
matching Requires in libibverbs-utils, which I like...
Doug, any thoughts from you on this front?
> > > > +install -D -m0644 Documentation/{ibacm,ibsrpdm,libibcm,libibverbs,librdmacm,rxe}.md %{buildroot}/%{_docdir}/%{name}-%{version}/
> > > > +install -D -m0644 README.md %{buildroot}/%{_docdir}/%{name}-%{version}/
> > >
> > > I guess README should go in that patch I sent you..
> >
> > D'oh, yeah, I'll mix that in here locally.
>
> I think you can just add ../README.md to the
> Documentation/CMakeFile.txt
First cut, I added a Files line to the root CMakeLists.txt. Would the
above be preferred over that? Both seem to work.
> > > > +%{buildroot}/%{_bindir}/ib_acme -D . -O
> > > > +# Fixup a multilib conflict in ibacm_opts.cfg:
> > > > +sed -i -e '/^# Specifies the directory of the provider libraries$/ a\
> > > > +# Use /usr/lib64/ibacm on 64bit, /usr/lib/libacm on 32bit.
> > > > +' -e 's%^\(# provider_lib_path
> > > > /usr/\)lib\(64\)\?/ibacm$%\1lib64/ibacm%' ibacm_opts.cfg
> > >
> > > Hum? I'm pretty sure this is basically fixed in the code now, was done here
> > > 5eebdb9baaaae420a4bb16e586a96807823916a0
> > >
> > > Adjusting the comment like that doesn't really make sense, the acm
> > > daemon has a fixed endianness and looks in a single place to load the
> > > plugin. If someone wants to use 32 bit plugins they have to install
> > > the 32 bit acm daemon, which would have the 32 bit path in the sample
> > > conf file...
> >
> > This was just copy and paste from our existing libacm spec, might well be
> > something that was fixed and I just missed it.
>
> It looks OK as-is to me, I'd drop it now.
Dropped, things do look fine w/o that.
> > They weren't getting installed, so I threw that hack in, meant to
> > actually say something about that, got lost in the shuffle. I'll
> > double-check if they're still not getting installed... Okay, they're
> > getting installed now. May have just been a transient error with an
> > earlier tarball, or something else I did wrong. I'll drop that bit.
>
> How strange.. FWIW, the tarball should be built with git-archive ..
There was a ton of churn from what I was doing, I blame me. ;) All good
now, and yes, creating tarballs with git archive:
$ git archive --prefix=rdma-core-11/ -o ~/rpmbuild/SOURCES/rdma-core-11.tgz --format tar.gz HEAD
--
Jarod Wilson
jarod-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: SQ overflow seen running isert traffic
From: Steve Wise @ 2016-10-18 14:34 UTC (permalink / raw)
To: 'Sagi Grimberg', 'Potnuri Bharat Teja'
Cc: target-devel-u79uwXL29TY76Z2rM5mHXA, nab-IzHhD5pYlfBP7FQvKIMDCQ,
linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <ed7ebb39-be81-00b3-ef23-3f4c0e3afbb1-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
>
> > I tried out this change and it works fine with iwarp. I dont see SQ
> > overflow. Apparently we have increased the sq too big to overflow. I am
going
> > to let it run with higher workloads for longer time, to see if it holds
good.
>
> Actually on second thought, this patch is an overkill. Effectively we
> now set:
>
> MAX_CMD=266
> and max_rdma_ctx=128 so together we take 394 which seems to too much.
>
> If we go by the scheme of 1 rdma + 1 send for each IO we need:
> - 128 sends
> - 128 rdmas
> - 10 miscs
>
> so this gives 266.
>
> Perhaps this is due to the fact that iWARP needs to register memory for
> rdma reads as well? (and also rdma writes > 128k for chelsio HW right?)
>
iWARP definitely needs to register memory for the target of reads, due to
REMOTE_WRITE requirement for the protocol. The source of a write doesn't need
to register memory, but the SGE depth can cause multiple WRITE WRs to be
required to service the IO. And in theory there should be some threshold where
it might be better performance-wise to do a memory register + 1 WRITE vs X
WRITEs.
As you mentioned, the RW API should account for this, but perhaps it is still
off some. Bharat, have a look into the RDMA-RW API and let us see if we can
figure out if the additional SQ depth it adds is sufficient.
> What is the workload you are running? with immediatedata enabled you
> should issue reg+rdma_read+send only for writes > 8k.
>
> Does this happen when you run only reads for example?
>
> I guess its time to get the sq accounting into shape...
So to sum up - 2 issues:
1) we believe the iSER + RW API correctly sizes the SQ, yet we're seeing SQ
overflows. So the SQ sizing needs more investigation.
2) if the SQ is full, then the iSER/target code is supposed to resubmit. And
apparently that isn't working.
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 04/10] mm: replace get_user_pages_locked() write/force parameters with gup_flags
From: Lorenzo Stoakes @ 2016-10-18 13:56 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, Linus Torvalds, Hugh Dickins, Dave Hansen, Rik van Riel,
Mel Gorman, Andrew Morton, adi-buildroot-devel, ceph-devel,
dri-devel, intel-gfx, kvm, linux-alpha, linux-arm-kernel,
linux-cris-kernel, linux-fbdev, linux-fsdevel, linux-ia64,
linux-kernel, linux-media, linux-mips, linux-rdma, linux-s390,
linux-samsung-soc
In-Reply-To: <20161018125425.GD29967@quack2.suse.cz>
On Tue, Oct 18, 2016 at 02:54:25PM +0200, Jan Kara wrote:
> > @@ -1282,7 +1282,7 @@ long get_user_pages(unsigned long start, unsigned long nr_pages,
> > int write, int force, struct page **pages,
> > struct vm_area_struct **vmas);
> > long get_user_pages_locked(unsigned long start, unsigned long nr_pages,
> > - int write, int force, struct page **pages, int *locked);
> > + unsigned int gup_flags, struct page **pages, int *locked);
>
> Hum, the prototype is inconsistent with e.g. __get_user_pages_unlocked()
> where gup_flags come after **pages argument. Actually it makes more sense
> to have it before **pages so that input arguments come first and output
> arguments second but I don't care that much. But it definitely should be
> consistent...
It was difficult to decide quite how to arrange parameters as there was
inconsitency with regards to parameter ordering already - for example
__get_user_pages() places its flags argument before pages whereas, as you note,
__get_user_pages_unlocked() puts them afterwards.
I ended up compromising by trying to match the existing ordering of the function
as much as I could by replacing write, force pairs with gup_flags in the same
location (with the exception of get_user_pages_unlocked() which I felt should
match __get_user_pages_unlocked() in signature) or if there was already a
gup_flags parameter as in the case of __get_user_pages_unlocked() I simply
removed the write, force pair and left the flags as the last parameter.
I am happy to rearrange parameters as needed, however I am not sure if it'd be
worthwhile for me to do so (I am keen to try to avoid adding too much noise here
:)
If we were to rearrange parameters for consistency I'd suggest adjusting
__get_user_pages_unlocked() to put gup_flags before pages and do the same with
get_user_pages_unlocked(), let me know what you think.
^ permalink raw reply
* Re: SQ overflow seen running isert traffic
From: Sagi Grimberg @ 2016-10-18 13:17 UTC (permalink / raw)
To: Potnuri Bharat Teja; +Cc: Steve Wise, target-devel, nab, linux-rdma
In-Reply-To: <20161018112801.GA3117@chelsio.com>
> I tried out this change and it works fine with iwarp. I dont see SQ
> overflow. Apparently we have increased the sq too big to overflow. I am going
> to let it run with higher workloads for longer time, to see if it holds good.
Actually on second thought, this patch is an overkill. Effectively we
now set:
MAX_CMD=266
and max_rdma_ctx=128 so together we take 394 which seems to too much.
If we go by the scheme of 1 rdma + 1 send for each IO we need:
- 128 sends
- 128 rdmas
- 10 miscs
so this gives 266.
Perhaps this is due to the fact that iWARP needs to register memory for
rdma reads as well? (and also rdma writes > 128k for chelsio HW right?)
What is the workload you are running? with immediatedata enabled you
should issue reg+rdma_read+send only for writes > 8k.
Does this happen when you run only reads for example?
I guess its time to get the sq accounting into shape...
^ permalink raw reply
* Re: [PATCH 04/10] mm: replace get_user_pages_locked() write/force parameters with gup_flags
From: Jan Kara @ 2016-10-18 12:54 UTC (permalink / raw)
To: Lorenzo Stoakes
Cc: linux-mips, linux-fbdev, Jan Kara, kvm, linux-sh, Dave Hansen,
dri-devel, linux-mm, netdev, sparclinux, linux-ia64, linux-s390,
linux-samsung-soc, linux-scsi, linux-rdma, x86, Hugh Dickins,
linux-media, Rik van Riel, intel-gfx, adi-buildroot-devel,
ceph-devel, linux-arm-kernel, linux-cris-kernel, Linus Torvalds,
linuxppc-dev, linux-kernel, linux-security-module, linux-alpha
In-Reply-To: <20161013002020.3062-5-lstoakes@gmail.com>
On Thu 13-10-16 01:20:14, Lorenzo Stoakes wrote:
> This patch removes the write and force parameters from get_user_pages_locked()
> and replaces them with a gup_flags parameter to make the use of FOLL_FORCE
> explicit in callers as use of this flag can result in surprising behaviour (and
> hence bugs) within the mm subsystem.
>
> Signed-off-by: Lorenzo Stoakes <lstoakes@gmail.com>
> ---
> include/linux/mm.h | 2 +-
> mm/frame_vector.c | 8 +++++++-
> mm/gup.c | 12 +++---------
> mm/nommu.c | 5 ++++-
> 4 files changed, 15 insertions(+), 12 deletions(-)
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 6adc4bc..27ab538 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1282,7 +1282,7 @@ long get_user_pages(unsigned long start, unsigned long nr_pages,
> int write, int force, struct page **pages,
> struct vm_area_struct **vmas);
> long get_user_pages_locked(unsigned long start, unsigned long nr_pages,
> - int write, int force, struct page **pages, int *locked);
> + unsigned int gup_flags, struct page **pages, int *locked);
Hum, the prototype is inconsistent with e.g. __get_user_pages_unlocked()
where gup_flags come after **pages argument. Actually it makes more sense
to have it before **pages so that input arguments come first and output
arguments second but I don't care that much. But it definitely should be
consistent...
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply
* Re: [PATCH 03/10] mm: replace get_user_pages_unlocked() write/force parameters with gup_flags
From: Jan Kara @ 2016-10-18 12:50 UTC (permalink / raw)
To: Lorenzo Stoakes
Cc: linux-mips, linux-fbdev, Jan Kara, kvm, linux-sh, Dave Hansen,
dri-devel, linux-mm, netdev, sparclinux, linux-ia64, linux-s390,
linux-samsung-soc, linux-scsi, linux-rdma, x86, Hugh Dickins,
linux-media, Rik van Riel, intel-gfx, adi-buildroot-devel,
ceph-devel, linux-arm-kernel, linux-cris-kernel, Linus Torvalds,
linuxppc-dev, linux-kernel, linux-security-module, linux-alpha
In-Reply-To: <20161013002020.3062-4-lstoakes@gmail.com>
On Thu 13-10-16 01:20:13, Lorenzo Stoakes wrote:
> This patch removes the write and force parameters from get_user_pages_unlocked()
> and replaces them with a gup_flags parameter to make the use of FOLL_FORCE
> explicit in callers as use of this flag can result in surprising behaviour (and
> hence bugs) within the mm subsystem.
>
> Signed-off-by: Lorenzo Stoakes <lstoakes@gmail.com>
Looks good. You can add:
Reviewed-by: Jan Kara <jack@suse.cz>
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply
* Re: [PATCH 02/10] mm: remove write/force parameters from __get_user_pages_unlocked()
From: Jan Kara @ 2016-10-18 12:46 UTC (permalink / raw)
To: Lorenzo Stoakes
Cc: linux-mips, linux-fbdev, Jan Kara, kvm, linux-sh, Dave Hansen,
dri-devel, linux-mm, netdev, sparclinux, linux-ia64, linux-s390,
linux-samsung-soc, linux-scsi, linux-rdma, x86, Hugh Dickins,
linux-media, Rik van Riel, intel-gfx, adi-buildroot-devel,
ceph-devel, linux-arm-kernel, linux-cris-kernel, Linus Torvalds,
linuxppc-dev, linux-kernel, linux-security-module, linux-alpha
In-Reply-To: <20161013002020.3062-3-lstoakes@gmail.com>
On Thu 13-10-16 01:20:12, Lorenzo Stoakes wrote:
> This patch removes the write and force parameters from
> __get_user_pages_unlocked() to make the use of FOLL_FORCE explicit in callers as
> use of this flag can result in surprising behaviour (and hence bugs) within the
> mm subsystem.
>
> Signed-off-by: Lorenzo Stoakes <lstoakes@gmail.com>
The patch looks good. You can add:
Reviewed-by: Jan Kara <jack@suse.cz>
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply
* Re: [PATCH 01/10] mm: remove write/force parameters from __get_user_pages_locked()
From: Jan Kara @ 2016-10-18 12:43 UTC (permalink / raw)
To: Lorenzo Stoakes
Cc: linux-mips, linux-fbdev, Jan Kara, kvm, linux-sh, Dave Hansen,
dri-devel, linux-mm, netdev, sparclinux, linux-ia64, linux-s390,
linux-samsung-soc, linux-scsi, linux-rdma, x86, Hugh Dickins,
linux-media, Rik van Riel, intel-gfx, adi-buildroot-devel,
ceph-devel, linux-arm-kernel, linux-cris-kernel, Linus Torvalds,
linuxppc-dev, linux-kernel, linux-security-module, linux-alpha
In-Reply-To: <20161013002020.3062-2-lstoakes@gmail.com>
On Thu 13-10-16 01:20:11, Lorenzo Stoakes wrote:
> This patch removes the write and force parameters from __get_user_pages_locked()
> to make the use of FOLL_FORCE explicit in callers as use of this flag can result
> in surprising behaviour (and hence bugs) within the mm subsystem.
>
> Signed-off-by: Lorenzo Stoakes <lstoakes@gmail.com>
Looks good. You can add:
Reviewed-by: Jan Kara <jack@suse.cz>
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply
* Re: SQ overflow seen running isert traffic
From: Potnuri Bharat Teja @ 2016-10-18 11:28 UTC (permalink / raw)
To: Sagi Grimberg; +Cc: Steve Wise, target-devel, nab, linux-rdma
In-Reply-To: <c4ee3313-9e67-f30a-3679-4c9afab79ac9@grimberg.me>
On Tuesday, October 10/18/16, 2016 at 11:04:00 +0300, Sagi Grimberg wrote:
> Hey Steve and Baharat,
>
> >Hey Sagi, I'm looking at isert_create_qp() and it appears to not be correctly
> >sizing the SQ:
> >...
> >#define ISERT_QP_MAX_REQ_DTOS (ISCSI_DEF_XMIT_CMDS_MAX + \
> > ISERT_MAX_TX_MISC_PDUS + \
> > ISERT_MAX_RX_MISC_PDUS)
> >...
> > attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS + 1;
> > attr.cap.max_recv_wr = ISERT_QP_MAX_RECV_DTOS + 1;
> >...
> >
> >I think above snipit assumes a DTO consumes exactly one WR/WQE in the SQ. But
> >the DTO can be broken into multiple WRs to handle REG_MRs, multiple WRITE or
> >READ WRs due to limits on local sge depths target sge depths, etc. Yes? Or am
> >I all wet? Or perhaps isert doesn't require the SQ to be the max possible
> >because it flow controls the DTO submissions?
>
> I think you are correct.
>
> On my test devices, I didn't see that multiple WRs has had any effect
> becuase:
> 1. My test devices usually give next power of 2 (256)
> 2. workloads that involved multiple rdma operations never stressed the
> system enough to get the queues full.
>
> Now, in iWARP for non-immediate writes we'll need more than a single
> wr per IO (I think the SQ size is expanded with the new rdma RW API
> which implicitly increases with attr.cap.max_rdma_ctxs).
Yes, rdma RW api factors the attr.cap.max_rdma_ctxs based on attr Flags.
>
> But I do agree that we need to take into account that each IO needs
> at least 2 WRs (one for rdma and one for send).
>
> So a temp bandage would be:
> --
> diff --git a/drivers/infiniband/ulp/isert/ib_isert.h
> b/drivers/infiniband/ulp/isert/ib_isert.h
> index fc791efe3a10..81afb95aeea9 100644
> --- a/drivers/infiniband/ulp/isert/ib_isert.h
> +++ b/drivers/infiniband/ulp/isert/ib_isert.h
> @@ -54,8 +54,14 @@
>
> #define ISERT_MIN_POSTED_RX (ISCSI_DEF_XMIT_CMDS_MAX >> 2)
>
> -#define ISERT_QP_MAX_REQ_DTOS (ISCSI_DEF_XMIT_CMDS_MAX + \
> - ISERT_MAX_TX_MISC_PDUS + \
> +/*
> + * Max QP send work requests consist of:
> + * - RDMA + SEND for each iscsi IO
> + * - iscsi misc TX pdus
> + * - iscsi misc RX response pdus
> + */
> +#define ISERT_QP_MAX_REQ_DTOS ((ISCSI_DEF_XMIT_CMDS_MAX * 2 ) + \
> + ISERT_MAX_TX_MISC_PDUS + \
> ISERT_MAX_RX_MISC_PDUS)
>
> #define ISER_RX_PAD_SIZE (ISCSI_DEF_MAX_RECV_SEG_LEN + 4096 - \
> --
>
I tried out this change and it works fine with iwarp. I dont see SQ
overflow. Apparently we have increased the sq too big to overflow. I am going
to let it run with higher workloads for longer time, to see if it holds good.
> But we do need to track the SQ overflow and queue a retransmit work when
> we don't have enough available SQ slots..
Agreed, iscsi-target (LIO in our case)expects failure to be returned by
overlying modules Or it could be for tcp as it handling is different to
that of iser.
It queues the WR and schedules it for repost incase
of post failures with ENOMEM/EAGAIN. Shall send the necessary change if
it is needed.
Thanks,
Bharat.
>
> Thoughts?
^ permalink raw reply
* 【For help】 configure crossbar build tool in CMakelist.txt
From: oulijun @ 2016-10-18 10:55 UTC (permalink / raw)
To: linux-rdma; +Cc: Linuxarm
Hi, all
I am working for debugging userspace code of hns in rdma-core.git.
I have met a cmake question now. please give a help for me.
if I use crossbar build tool aarch64-linux-gnu-gcc for building the directory(provider/hns), what i should do it ?
My modification currently according to the others as fllows:
in the file : provider/hns/CMakelist.txt
set(CMAKE_C_COMPILER /opt/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux/bin/aarch64-linux-gnu-gcc)
but the modification is fail
-- The C compiler identification is GNU 4.8.4
-- Check for working C compiler using: Ninja
-- Check for working C compiler using: Ninja -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- cmake_share:
-- GNU:
-- BUILD_BIN_DIR:/home/ubuntu/rdma-core/build/bin
-- CMAKE_INSTALL: /usr/local/var/run
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.26")
-- rdma_build: Debug;Release;RelWithDebInfo;MinSizeRel
-- BUILD_INCLUDE_DIR:/home/ubuntu/rdma-core/build/include
-- Performing Test SUPPORTS_GNU99
-- Performing Test SUPPORTS_GNU99 - Success
-- Performing Test HAVE_C_WARNINGS
-- Performing Test HAVE_C_WARNINGS - Success
-- Performing Test HAVE_C_WMISSING_PROTOTYPES
-- Performing Test HAVE_C_WMISSING_PROTOTYPES - Success
-- Performing Test HAVE_C_WMISSING_DECLARATIONS
-- Performing Test HAVE_C_WMISSING_DECLARATIONS - Success
-- Performing Test HAVE_C_WWRITE_STRINGS
-- Performing Test HAVE_C_WWRITE_STRINGS - Success
-- Performing Test HAVE_C_WFORMAT_2
-- Performing Test HAVE_C_WFORMAT_2 - Success
-- Performing Test HAVE_C_WORKING_SHADOW
-- Performing Test HAVE_C_WORKING_SHADOW - Success
-- Performing Test HAVE_C_WORKING_MISSING_FIELD_INITIALIZERS
-- Performing Test HAVE_C_WORKING_MISSING_FIELD_INITIALIZERS - Failed
-- Performing Test HAVE_C_WNO_MISSING_FIELD_INITIALIZERS
-- Performing Test HAVE_C_WNO_MISSING_FIELD_INITIALIZERS - Success
-- Performing Test HAVE_NO_STRICT_ALIASING
-- Performing Test HAVE_NO_STRICT_ALIASING - Success
-- Performing Test HAS_CLOEXEC
-- Performing Test HAS_CLOEXEC - Success
-- Performing Test HAVE_FUNC_ATTRIBUTE_ALWAYS_INLINE
-- Performing Test HAVE_FUNC_ATTRIBUTE_ALWAYS_INLINE - Success
-- Performing Test SUPPORTS_AS_NEEDED
-- Performing Test SUPPORTS_AS_NEEDED - Success
-- Performing Test SUPPORTS_NO_UNDEFINED
-- Performing Test SUPPORTS_NO_UNDEFINED - Success
-- Performing Test _LDSYMVER_SUCCESS
-- Performing Test _LDSYMVER_SUCCESS - Success
-- Found LDSymVer: GNU
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- checking for modules 'libnl-3.0;libnl-route-3.0'
-- found libnl-3.0, version 3.2.21
-- found libnl-route-3.0, version 3.2.21
-- Performing Test HAVE_WORKING_IF_H
-- Performing Test HAVE_WORKING_IF_H - Failed
-- Check size of long
-- Check size of long - done
-- Looking for rdma/rdma_user_rxe.h
-- Looking for rdma/rdma_user_rxe.h - not found
-- Looking for valgrind/memcheck.h
-- Looking for valgrind/memcheck.h - found
-- Looking for valgrind/drd.h
-- Looking for valgrind/drd.h - found
-- Performing Test LIBC_HAS_LIBRT
-- Performing Test LIBC_HAS_LIBRT - Success
-- Performing Test HAVE_C_WSTRICT_PROTOTYPES
-- Performing Test HAVE_C_WSTRICT_PROTOTYPES - Success
-- Performing Test HAVE_C_WOLD_STYLE_DEFINITION
-- Performing Test HAVE_C_WOLD_STYLE_DEFINITION - Success
-- Performing Test HAVE_C_WREDUNDANT_DECLS
-- Performing Test HAVE_C_WREDUNDANT_DECLS - Failed
-- c_complier: /opt/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux/bin/aarch64-linux-gnu-gcc
-- Missing Optional Items:
-- netlink/route/link.h and net/if.h NOT co-includable (old headers)
-- rdma/rdma_user_rxe.h NOT found (old system kernel headers)
-- -Wmissing-field-initializers does NOT work
-- -Wredundant-decls does NOT work
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ubuntu/rdma-core/build
[115/164] Building C object providers/hns/CMakeFiles/hns-rdmav2.dir/hns_roce_u_hw_v1.c.o
FAILED: /usr/bin/cc -Dhns_rdmav2_EXPORTS -std=gnu99 -Wall -Wextra -Wno-sign-compare -Wno-unused-parameter -Wmissing-prototypes -Wmissing-declarations -Wwrite-strings -Wformat=2 -Wshadow -Wno-missing-field-initializers -Wstrict-prototypes -Wold-style-definition -O2 -g -fPIC -Iinclude -I/usr/include/libnl3 -MMD -MT providers/hns/CMakeFiles/hns-rdmav2.dir/hns_roce_u_hw_v1.c.o -MF "providers/hns/CMakeFiles/hns-rdmav2.dir/hns_roce_u_hw_v1.c.o.d" -o providers/hns/CMakeFiles/hns-rdmav2.dir/hns_roce_u_hw_v1.c.o -c ../providers/hns/hns_roce_u_hw_v1.c
../providers/hns/hns_roce_u_hw_v1.c: Assembler messages:
../providers/hns/hns_roce_u_hw_v1.c:830: Error: number of operands mismatch for `ds'
../providers/hns/hns_roce_u_hw_v1.c:264: Error: number of operands mismatch for `ds'
../providers/hns/hns_roce_u_hw_v1.c:413: Error: number of operands mismatch for `ds'
../providers/hns/hns_roce_u_hw_v1.c:586: Error: number of operands mismatch for `ds'
../providers/hns/hns_roce_u_hw_v1.c:630: Error: number of operands mismatch for `ds'
[115/164] Building C object ibacm/CMakeFiles/ibacm.dir/src/acm.c.o
ninja: build stopped: subcommand failed.
.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 03/28] [v2] infiniband: shut up a maybe-uninitialized warning
From: Haggai Eran @ 2016-10-18 10:32 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Doug Ledford, Linus Torvalds, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
Sean Hefty, Hal Rosenstock, Matan Barak, Leon Romanovsky,
Sagi Grimberg, Bart Van Assche, Alex Vesker, Guy Shapiro,
linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <9469876.gij7f9Mh1b@wuerfel>
On 10/18/2016 1:18 PM, Arnd Bergmann wrote:
> On Tuesday, October 18, 2016 9:47:31 AM CEST Haggai Eran wrote:
>> On 10/18/2016 1:05 AM, Arnd Bergmann wrote:
>>> @@ -1309,7 +1311,7 @@ static bool validate_net_dev(struct net_device *net_dev,
>>> static struct net_device *cma_get_net_dev(struct ib_cm_event *ib_event,
>>> const struct cma_req_info *req)
>>> {
>>> - struct sockaddr_storage listen_addr_storage, src_addr_storage;
>>> + struct sockaddr_storage listen_addr_storage = {}, src_addr_storage = {};
>>
>> Doesn't this still translate to an extra initialization that Doug was
>> worried about?
>
> Thanks for spotting this. I must have screwed up while rebasing the patch
> at some point, this one change should not be there, the other changes by
> themselves sufficiently address the warning.
Okay, other than this the patch looks good to me.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 03/28] [v2] infiniband: shut up a maybe-uninitialized warning
From: Arnd Bergmann @ 2016-10-18 10:18 UTC (permalink / raw)
To: Haggai Eran
Cc: Doug Ledford, Linus Torvalds, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
Sean Hefty, Hal Rosenstock, Matan Barak, Leon Romanovsky,
Sagi Grimberg, Bart Van Assche, Alex Vesker, Guy Shapiro,
linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <33302790-0a4c-e2b3-868d-3e7dadbd3c07-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
On Tuesday, October 18, 2016 9:47:31 AM CEST Haggai Eran wrote:
> On 10/18/2016 1:05 AM, Arnd Bergmann wrote:
> > @@ -1309,7 +1311,7 @@ static bool validate_net_dev(struct net_device *net_dev,
> > static struct net_device *cma_get_net_dev(struct ib_cm_event *ib_event,
> > const struct cma_req_info *req)
> > {
> > - struct sockaddr_storage listen_addr_storage, src_addr_storage;
> > + struct sockaddr_storage listen_addr_storage = {}, src_addr_storage = {};
>
> Doesn't this still translate to an extra initialization that Doug was
> worried about?
Thanks for spotting this. I must have screwed up while rebasing the patch
at some point, this one change should not be there, the other changes by
themselves sufficiently address the warning.
Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: RQ overflow seen running isert traffic
From: Sagi Grimberg @ 2016-10-18 8:04 UTC (permalink / raw)
To: Steve Wise, 'Potnuri Bharat Teja'
Cc: target-devel-u79uwXL29TY76Z2rM5mHXA, nab-IzHhD5pYlfBP7FQvKIMDCQ,
linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <021001d228a4$6cd6a6c0$4683f440$@opengridcomputing.com>
Hey Steve and Baharat,
> Hey Sagi, I'm looking at isert_create_qp() and it appears to not be correctly
> sizing the SQ:
> ...
> #define ISERT_QP_MAX_REQ_DTOS (ISCSI_DEF_XMIT_CMDS_MAX + \
> ISERT_MAX_TX_MISC_PDUS + \
> ISERT_MAX_RX_MISC_PDUS)
> ...
> attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS + 1;
> attr.cap.max_recv_wr = ISERT_QP_MAX_RECV_DTOS + 1;
> ...
>
> I think above snipit assumes a DTO consumes exactly one WR/WQE in the SQ. But
> the DTO can be broken into multiple WRs to handle REG_MRs, multiple WRITE or
> READ WRs due to limits on local sge depths target sge depths, etc. Yes? Or am
> I all wet? Or perhaps isert doesn't require the SQ to be the max possible
> because it flow controls the DTO submissions?
I think you are correct.
On my test devices, I didn't see that multiple WRs has had any effect
becuase:
1. My test devices usually give next power of 2 (256)
2. workloads that involved multiple rdma operations never stressed the
system enough to get the queues full.
Now, in iWARP for non-immediate writes we'll need more than a single
wr per IO (I think the SQ size is expanded with the new rdma RW API
which implicitly increases with attr.cap.max_rdma_ctxs).
But I do agree that we need to take into account that each IO needs
at least 2 WRs (one for rdma and one for send).
So a temp bandage would be:
--
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h
b/drivers/infiniband/ulp/isert/ib_isert.h
index fc791efe3a10..81afb95aeea9 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -54,8 +54,14 @@
#define ISERT_MIN_POSTED_RX (ISCSI_DEF_XMIT_CMDS_MAX >> 2)
-#define ISERT_QP_MAX_REQ_DTOS (ISCSI_DEF_XMIT_CMDS_MAX + \
- ISERT_MAX_TX_MISC_PDUS + \
+/*
+ * Max QP send work requests consist of:
+ * - RDMA + SEND for each iscsi IO
+ * - iscsi misc TX pdus
+ * - iscsi misc RX response pdus
+ */
+#define ISERT_QP_MAX_REQ_DTOS ((ISCSI_DEF_XMIT_CMDS_MAX * 2 ) + \
+ ISERT_MAX_TX_MISC_PDUS + \
ISERT_MAX_RX_MISC_PDUS)
#define ISER_RX_PAD_SIZE (ISCSI_DEF_MAX_RECV_SEG_LEN + 4096 - \
--
But we do need to track the SQ overflow and queue a retransmit work when
we don't have enough available SQ slots..
Thoughts?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: iscsi_trx going into D state
From: Nicholas A. Bellinger @ 2016-10-18 7:52 UTC (permalink / raw)
To: Robert LeBlanc; +Cc: Zhu Lingshan, linux-rdma, linux-scsi
In-Reply-To: <1476774332.8490.43.camel@haakon3.risingtidesystems.com>
On Tue, 2016-10-18 at 00:05 -0700, Nicholas A. Bellinger wrote:
> Hello Robert, Zhu & Co,
>
> Thanks for your detailed bug report. Comments inline below.
>
> On Mon, 2016-10-17 at 22:42 -0600, Robert LeBlanc wrote:
> > Sorry I forget that Android has an aversion to plain text emails.
> >
> > If we can provide any information to help, let us know. We are willing
> > to patch in more debug statements or whatever you think might help.
> > Today has been a difficult day. Thanks for looking into it, I tried
> > looking at it, but it is way over my head.
> >
> > ----------------
> > Robert LeBlanc
> > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
> >
> >
> > On Mon, Oct 17, 2016 at 9:06 PM, Zhu Lingshan <lszhu@suse.com> wrote:
> > > Hi Robert,
> > >
> > > I think the reason why you can not logout the targets is that iscsi_np in D
> > > status. I think the patches fixed something, but it seems to be more than
> > > one code path can trigger these similar issues. as you can see, there are
> > > several call stacks, I am still working on it. Actually in my environment I
> > > see there is another call stack not listed in your mail....
> > >
> > > Thanks,
> > > BR
> > > Zhu Lingshan
> > >
> > >
> > > On 10/18/2016 03:11 AM, Robert LeBlanc wrote:
> > >>
> > >> Sorry hit send too soon.
> > >>
> > >> In addition, on the client we see:
> > >> # ps -aux | grep D | grep kworker
> > >> root 5583 0.0 0.0 0 0 ? D 11:55 0:03
> > >> [kworker/11:0]
> > >> root 7721 0.1 0.0 0 0 ? D 12:00 0:04
> > >> [kworker/4:25]
> > >> root 10877 0.0 0.0 0 0 ? D 09:27 0:00
> > >> [kworker/22:1]
> > >> root 11246 0.0 0.0 0 0 ? D 10:28 0:00
> > >> [kworker/30:2]
> > >> root 14034 0.0 0.0 0 0 ? D 12:20 0:02
> > >> [kworker/19:15]
> > >> root 14048 0.0 0.0 0 0 ? D 12:20 0:00
> > >> [kworker/16:0]
> > >> root 15871 0.0 0.0 0 0 ? D 12:25 0:00
> > >> [kworker/13:0]
> > >> root 17442 0.0 0.0 0 0 ? D 12:28 0:00
> > >> [kworker/9:1]
> > >> root 17816 0.0 0.0 0 0 ? D 12:30 0:00
> > >> [kworker/11:1]
> > >> root 18744 0.0 0.0 0 0 ? D 12:32 0:00
> > >> [kworker/10:2]
> > >> root 19060 0.0 0.0 0 0 ? D 12:32 0:00
> > >> [kworker/29:0]
> > >> root 21748 0.0 0.0 0 0 ? D 12:40 0:00
> > >> [kworker/21:0]
> > >> root 21967 0.0 0.0 0 0 ? D 12:40 0:00
> > >> [kworker/22:0]
> > >> root 21978 0.0 0.0 0 0 ? D 12:40 0:00
> > >> [kworker/22:2]
> > >> root 22024 0.0 0.0 0 0 ? D 12:40 0:00
> > >> [kworker/22:4]
> > >> root 22035 0.0 0.0 0 0 ? D 12:40 0:00
> > >> [kworker/22:5]
> > >> root 22060 0.0 0.0 0 0 ? D 12:40 0:00
> > >> [kworker/16:1]
> > >> root 22282 0.0 0.0 0 0 ? D 12:41 0:00
> > >> [kworker/26:0]
> > >> root 22362 0.0 0.0 0 0 ? D 12:42 0:00
> > >> [kworker/18:9]
> > >> root 22426 0.0 0.0 0 0 ? D 12:42 0:00
> > >> [kworker/16:3]
> > >> root 23298 0.0 0.0 0 0 ? D 12:43 0:00
> > >> [kworker/12:1]
> > >> root 23302 0.0 0.0 0 0 ? D 12:43 0:00
> > >> [kworker/12:5]
> > >> root 24264 0.0 0.0 0 0 ? D 12:46 0:00
> > >> [kworker/30:1]
> > >> root 24271 0.0 0.0 0 0 ? D 12:46 0:00
> > >> [kworker/14:8]
> > >> root 24441 0.0 0.0 0 0 ? D 12:47 0:00
> > >> [kworker/9:7]
> > >> root 24443 0.0 0.0 0 0 ? D 12:47 0:00
> > >> [kworker/9:9]
> > >> root 25005 0.0 0.0 0 0 ? D 12:48 0:00
> > >> [kworker/30:3]
> > >> root 25158 0.0 0.0 0 0 ? D 12:49 0:00
> > >> [kworker/9:12]
> > >> root 26382 0.0 0.0 0 0 ? D 12:52 0:00
> > >> [kworker/13:2]
> > >> root 26453 0.0 0.0 0 0 ? D 12:52 0:00
> > >> [kworker/21:2]
> > >> root 26724 0.0 0.0 0 0 ? D 12:53 0:00
> > >> [kworker/19:1]
> > >> root 28400 0.0 0.0 0 0 ? D 05:20 0:00
> > >> [kworker/25:1]
> > >> root 29552 0.0 0.0 0 0 ? D 11:40 0:00
> > >> [kworker/17:1]
> > >> root 29811 0.0 0.0 0 0 ? D 11:40 0:00
> > >> [kworker/7:10]
> > >> root 31903 0.0 0.0 0 0 ? D 11:43 0:00
> > >> [kworker/26:1]
> > >>
> > >> And all of the processes have this stack:
> > >> [<ffffffffa0727ed5>] iser_release_work+0x25/0x60 [ib_iser]
> > >> [<ffffffff8109633f>] process_one_work+0x14f/0x400
> > >> [<ffffffff81096bb4>] worker_thread+0x114/0x470
> > >> [<ffffffff8109c6f8>] kthread+0xd8/0xf0
> > >> [<ffffffff8172004f>] ret_from_fork+0x3f/0x70
> > >> [<ffffffffffffffff>] 0xffffffffffffffff
> > >>
> > >> We are not able to log out of the sessions in all cases. And have to
> > >> restart the box.
> > >>
> > >> iscsiadm -m session will show messages like:
> > >> iscsiadm: could not read session targetname: 5
> > >> iscsiadm: could not find session info for session100
> > >> iscsiadm: could not read session targetname: 5
> > >> iscsiadm: could not find session info for session101
> > >> iscsiadm: could not read session targetname: 5
> > >> iscsiadm: could not find session info for session103
> > >> ...
> > >>
> > >> I can't find any way to force iscsiadm to clean up these sessions
> > >> possibly due to tasks in D state.
> > >> ----------------
> > >> Robert LeBlanc
> > >> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
> > >>
> > >>
> > >> On Mon, Oct 17, 2016 at 10:32 AM, Robert LeBlanc <robert@leblancnet.us>
> > >> wrote:
> > >>>
> > >>> Some more info as we hit this this morning. We have volumes mirrored
> > >>> between two targets and we had one target on the kernel with the three
> > >>> patches mentioned in this thread [0][1][2] and the other was on a
> > >>> kernel without the patches. We decided that after a week and a half we
> > >>> wanted to get both targets on the same kernel so we rebooted the
> > >>> non-patched target. Within an hour we saw iSCSI in D state with the
> > >>> same stack trace so it seems that we are not hitting any of the
> > >>> WARN_ON lines. We are getting both iscsi_trx and iscsi_np both in D
> > >>> state, this time we have two iscsi_trx processes in D state. I don't
> > >>> know if stale sessions on the clients could be contributing to this
> > >>> issue (the target trying to close non-existent sessions??). This is on
> > >>> 4.4.23. Any more debug info we can throw at this problem to help?
> > >>>
> > >>> Thank you,
> > >>> Robert LeBlanc
> > >>>
> > >>> # ps aux | grep D | grep iscsi
> > >>> root 16525 0.0 0.0 0 0 ? D 08:50 0:00
> > >>> [iscsi_np]
> > >>> root 16614 0.0 0.0 0 0 ? D 08:50 0:00
> > >>> [iscsi_trx]
> > >>> root 16674 0.0 0.0 0 0 ? D 08:50 0:00
> > >>> [iscsi_trx]
> > >>>
> > >>> # for i in 16525 16614 16674; do echo $i; cat /proc/$i/stack; done
> > >>> 16525
> > >>> [<ffffffff814f0d5f>] iscsit_stop_session+0x19f/0x1d0
> > >>> [<ffffffff814e2516>] iscsi_check_for_session_reinstatement+0x1e6/0x270
> > >>> [<ffffffff814e4ed0>] iscsi_target_check_for_existing_instances+0x30/0x40
> > >>> [<ffffffff814e5020>] iscsi_target_do_login+0x140/0x640
> > >>> [<ffffffff814e63bc>] iscsi_target_start_negotiation+0x1c/0xb0
> > >>> [<ffffffff814e410b>] iscsi_target_login_thread+0xa9b/0xfc0
> > >>> [<ffffffff8109c748>] kthread+0xd8/0xf0
> > >>> [<ffffffff8172018f>] ret_from_fork+0x3f/0x70
> > >>> [<ffffffffffffffff>] 0xffffffffffffffff
> > >>> 16614
> > >>> [<ffffffff814cca79>] target_wait_for_sess_cmds+0x49/0x1a0
> > >>> [<ffffffffa064692b>] isert_wait_conn+0x1ab/0x2f0 [ib_isert]
> > >>> [<ffffffff814f0ef2>] iscsit_close_connection+0x162/0x870
> > >>> [<ffffffff814df9bf>] iscsit_take_action_for_connection_exit+0x7f/0x100
> > >>> [<ffffffff814f00a0>] iscsi_target_rx_thread+0x5a0/0xe80
> > >>> [<ffffffff8109c748>] kthread+0xd8/0xf0
> > >>> [<ffffffff8172018f>] ret_from_fork+0x3f/0x70
> > >>> [<ffffffffffffffff>] 0xffffffffffffffff
> > >>> 16674
> > >>> [<ffffffff814cca79>] target_wait_for_sess_cmds+0x49/0x1a0
> > >>> [<ffffffffa064692b>] isert_wait_conn+0x1ab/0x2f0 [ib_isert]
> > >>> [<ffffffff814f0ef2>] iscsit_close_connection+0x162/0x870
> > >>> [<ffffffff814df9bf>] iscsit_take_action_for_connection_exit+0x7f/0x100
> > >>> [<ffffffff814f00a0>] iscsi_target_rx_thread+0x5a0/0xe80
> > >>> [<ffffffff8109c748>] kthread+0xd8/0xf0
> > >>> [<ffffffff8172018f>] ret_from_fork+0x3f/0x70
> > >>> [<ffffffffffffffff>] 0xffffffffffffffff
> > >>>
> > >>>
> > >>> [0] https://www.spinics.net/lists/target-devel/msg13463.html
> > >>> [1] http://marc.info/?l=linux-scsi&m=147282568910535&w=2
> > >>> [2] http://www.spinics.net/lists/linux-scsi/msg100221.html
>
> The call chain above is iscsi session reinstatement driven by
> open-iscsi/iser resulting in target-core to sleep indefinitely, waiting
> for outstanding target-core backend driver se_cmd I/O to complete in
> order to make forward progress.
>
> Note, there is a v4.1+ se_cmd->cmd_kref reference leak bug for
> TMR ABORT_TASK during simultaneous target back-end I/O completion
> timeouts here:
>
> http://www.spinics.net/lists/target-devel/msg13530.html
>
> If you are actively observing TMR ABORT_TASK preceding the hung task
> timeout warnings above with v4.4.y + v4.2.y iser-target exports, then
> it's likely the same bug. Please apply the patch on your v4.x setup to
> verify.
>
> If no TMR ABORT_TASK timeouts + session reinstatements are occurring on
> your iser-target setup, then it is a separate bug.
>
To clarify a bit more..
Using a v4.1.26+ kernel with traditional iscsi-target exports and patch
in place, I can confirm iscsi-target is able to successfully invoke
configfs network portal group delete via syscall:
rmdir /sys/kernel/config/target/iscsi/$IQN/$TPGT/np/$IPv4:$PORT
after TMR ABORT_TASKs due to backend I/O timeout + iscsi session
reinstatement scenario have occurred.
^ permalink raw reply
* Re: iscsi_trx going into D state
From: Nicholas A. Bellinger @ 2016-10-18 7:05 UTC (permalink / raw)
To: Robert LeBlanc; +Cc: Zhu Lingshan, linux-rdma, linux-scsi
In-Reply-To: <CAANLjFpx1MRNiu2388pY+U0O5z9gw+OJBtpOof47Lb_DxaJSBQ@mail.gmail.com>
Hello Robert, Zhu & Co,
Thanks for your detailed bug report. Comments inline below.
On Mon, 2016-10-17 at 22:42 -0600, Robert LeBlanc wrote:
> Sorry I forget that Android has an aversion to plain text emails.
>
> If we can provide any information to help, let us know. We are willing
> to patch in more debug statements or whatever you think might help.
> Today has been a difficult day. Thanks for looking into it, I tried
> looking at it, but it is way over my head.
>
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
>
>
> On Mon, Oct 17, 2016 at 9:06 PM, Zhu Lingshan <lszhu@suse.com> wrote:
> > Hi Robert,
> >
> > I think the reason why you can not logout the targets is that iscsi_np in D
> > status. I think the patches fixed something, but it seems to be more than
> > one code path can trigger these similar issues. as you can see, there are
> > several call stacks, I am still working on it. Actually in my environment I
> > see there is another call stack not listed in your mail....
> >
> > Thanks,
> > BR
> > Zhu Lingshan
> >
> >
> > On 10/18/2016 03:11 AM, Robert LeBlanc wrote:
> >>
> >> Sorry hit send too soon.
> >>
> >> In addition, on the client we see:
> >> # ps -aux | grep D | grep kworker
> >> root 5583 0.0 0.0 0 0 ? D 11:55 0:03
> >> [kworker/11:0]
> >> root 7721 0.1 0.0 0 0 ? D 12:00 0:04
> >> [kworker/4:25]
> >> root 10877 0.0 0.0 0 0 ? D 09:27 0:00
> >> [kworker/22:1]
> >> root 11246 0.0 0.0 0 0 ? D 10:28 0:00
> >> [kworker/30:2]
> >> root 14034 0.0 0.0 0 0 ? D 12:20 0:02
> >> [kworker/19:15]
> >> root 14048 0.0 0.0 0 0 ? D 12:20 0:00
> >> [kworker/16:0]
> >> root 15871 0.0 0.0 0 0 ? D 12:25 0:00
> >> [kworker/13:0]
> >> root 17442 0.0 0.0 0 0 ? D 12:28 0:00
> >> [kworker/9:1]
> >> root 17816 0.0 0.0 0 0 ? D 12:30 0:00
> >> [kworker/11:1]
> >> root 18744 0.0 0.0 0 0 ? D 12:32 0:00
> >> [kworker/10:2]
> >> root 19060 0.0 0.0 0 0 ? D 12:32 0:00
> >> [kworker/29:0]
> >> root 21748 0.0 0.0 0 0 ? D 12:40 0:00
> >> [kworker/21:0]
> >> root 21967 0.0 0.0 0 0 ? D 12:40 0:00
> >> [kworker/22:0]
> >> root 21978 0.0 0.0 0 0 ? D 12:40 0:00
> >> [kworker/22:2]
> >> root 22024 0.0 0.0 0 0 ? D 12:40 0:00
> >> [kworker/22:4]
> >> root 22035 0.0 0.0 0 0 ? D 12:40 0:00
> >> [kworker/22:5]
> >> root 22060 0.0 0.0 0 0 ? D 12:40 0:00
> >> [kworker/16:1]
> >> root 22282 0.0 0.0 0 0 ? D 12:41 0:00
> >> [kworker/26:0]
> >> root 22362 0.0 0.0 0 0 ? D 12:42 0:00
> >> [kworker/18:9]
> >> root 22426 0.0 0.0 0 0 ? D 12:42 0:00
> >> [kworker/16:3]
> >> root 23298 0.0 0.0 0 0 ? D 12:43 0:00
> >> [kworker/12:1]
> >> root 23302 0.0 0.0 0 0 ? D 12:43 0:00
> >> [kworker/12:5]
> >> root 24264 0.0 0.0 0 0 ? D 12:46 0:00
> >> [kworker/30:1]
> >> root 24271 0.0 0.0 0 0 ? D 12:46 0:00
> >> [kworker/14:8]
> >> root 24441 0.0 0.0 0 0 ? D 12:47 0:00
> >> [kworker/9:7]
> >> root 24443 0.0 0.0 0 0 ? D 12:47 0:00
> >> [kworker/9:9]
> >> root 25005 0.0 0.0 0 0 ? D 12:48 0:00
> >> [kworker/30:3]
> >> root 25158 0.0 0.0 0 0 ? D 12:49 0:00
> >> [kworker/9:12]
> >> root 26382 0.0 0.0 0 0 ? D 12:52 0:00
> >> [kworker/13:2]
> >> root 26453 0.0 0.0 0 0 ? D 12:52 0:00
> >> [kworker/21:2]
> >> root 26724 0.0 0.0 0 0 ? D 12:53 0:00
> >> [kworker/19:1]
> >> root 28400 0.0 0.0 0 0 ? D 05:20 0:00
> >> [kworker/25:1]
> >> root 29552 0.0 0.0 0 0 ? D 11:40 0:00
> >> [kworker/17:1]
> >> root 29811 0.0 0.0 0 0 ? D 11:40 0:00
> >> [kworker/7:10]
> >> root 31903 0.0 0.0 0 0 ? D 11:43 0:00
> >> [kworker/26:1]
> >>
> >> And all of the processes have this stack:
> >> [<ffffffffa0727ed5>] iser_release_work+0x25/0x60 [ib_iser]
> >> [<ffffffff8109633f>] process_one_work+0x14f/0x400
> >> [<ffffffff81096bb4>] worker_thread+0x114/0x470
> >> [<ffffffff8109c6f8>] kthread+0xd8/0xf0
> >> [<ffffffff8172004f>] ret_from_fork+0x3f/0x70
> >> [<ffffffffffffffff>] 0xffffffffffffffff
> >>
> >> We are not able to log out of the sessions in all cases. And have to
> >> restart the box.
> >>
> >> iscsiadm -m session will show messages like:
> >> iscsiadm: could not read session targetname: 5
> >> iscsiadm: could not find session info for session100
> >> iscsiadm: could not read session targetname: 5
> >> iscsiadm: could not find session info for session101
> >> iscsiadm: could not read session targetname: 5
> >> iscsiadm: could not find session info for session103
> >> ...
> >>
> >> I can't find any way to force iscsiadm to clean up these sessions
> >> possibly due to tasks in D state.
> >> ----------------
> >> Robert LeBlanc
> >> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
> >>
> >>
> >> On Mon, Oct 17, 2016 at 10:32 AM, Robert LeBlanc <robert@leblancnet.us>
> >> wrote:
> >>>
> >>> Some more info as we hit this this morning. We have volumes mirrored
> >>> between two targets and we had one target on the kernel with the three
> >>> patches mentioned in this thread [0][1][2] and the other was on a
> >>> kernel without the patches. We decided that after a week and a half we
> >>> wanted to get both targets on the same kernel so we rebooted the
> >>> non-patched target. Within an hour we saw iSCSI in D state with the
> >>> same stack trace so it seems that we are not hitting any of the
> >>> WARN_ON lines. We are getting both iscsi_trx and iscsi_np both in D
> >>> state, this time we have two iscsi_trx processes in D state. I don't
> >>> know if stale sessions on the clients could be contributing to this
> >>> issue (the target trying to close non-existent sessions??). This is on
> >>> 4.4.23. Any more debug info we can throw at this problem to help?
> >>>
> >>> Thank you,
> >>> Robert LeBlanc
> >>>
> >>> # ps aux | grep D | grep iscsi
> >>> root 16525 0.0 0.0 0 0 ? D 08:50 0:00
> >>> [iscsi_np]
> >>> root 16614 0.0 0.0 0 0 ? D 08:50 0:00
> >>> [iscsi_trx]
> >>> root 16674 0.0 0.0 0 0 ? D 08:50 0:00
> >>> [iscsi_trx]
> >>>
> >>> # for i in 16525 16614 16674; do echo $i; cat /proc/$i/stack; done
> >>> 16525
> >>> [<ffffffff814f0d5f>] iscsit_stop_session+0x19f/0x1d0
> >>> [<ffffffff814e2516>] iscsi_check_for_session_reinstatement+0x1e6/0x270
> >>> [<ffffffff814e4ed0>] iscsi_target_check_for_existing_instances+0x30/0x40
> >>> [<ffffffff814e5020>] iscsi_target_do_login+0x140/0x640
> >>> [<ffffffff814e63bc>] iscsi_target_start_negotiation+0x1c/0xb0
> >>> [<ffffffff814e410b>] iscsi_target_login_thread+0xa9b/0xfc0
> >>> [<ffffffff8109c748>] kthread+0xd8/0xf0
> >>> [<ffffffff8172018f>] ret_from_fork+0x3f/0x70
> >>> [<ffffffffffffffff>] 0xffffffffffffffff
> >>> 16614
> >>> [<ffffffff814cca79>] target_wait_for_sess_cmds+0x49/0x1a0
> >>> [<ffffffffa064692b>] isert_wait_conn+0x1ab/0x2f0 [ib_isert]
> >>> [<ffffffff814f0ef2>] iscsit_close_connection+0x162/0x870
> >>> [<ffffffff814df9bf>] iscsit_take_action_for_connection_exit+0x7f/0x100
> >>> [<ffffffff814f00a0>] iscsi_target_rx_thread+0x5a0/0xe80
> >>> [<ffffffff8109c748>] kthread+0xd8/0xf0
> >>> [<ffffffff8172018f>] ret_from_fork+0x3f/0x70
> >>> [<ffffffffffffffff>] 0xffffffffffffffff
> >>> 16674
> >>> [<ffffffff814cca79>] target_wait_for_sess_cmds+0x49/0x1a0
> >>> [<ffffffffa064692b>] isert_wait_conn+0x1ab/0x2f0 [ib_isert]
> >>> [<ffffffff814f0ef2>] iscsit_close_connection+0x162/0x870
> >>> [<ffffffff814df9bf>] iscsit_take_action_for_connection_exit+0x7f/0x100
> >>> [<ffffffff814f00a0>] iscsi_target_rx_thread+0x5a0/0xe80
> >>> [<ffffffff8109c748>] kthread+0xd8/0xf0
> >>> [<ffffffff8172018f>] ret_from_fork+0x3f/0x70
> >>> [<ffffffffffffffff>] 0xffffffffffffffff
> >>>
> >>>
> >>> [0] https://www.spinics.net/lists/target-devel/msg13463.html
> >>> [1] http://marc.info/?l=linux-scsi&m=147282568910535&w=2
> >>> [2] http://www.spinics.net/lists/linux-scsi/msg100221.html
The call chain above is iscsi session reinstatement driven by
open-iscsi/iser resulting in target-core to sleep indefinitely, waiting
for outstanding target-core backend driver se_cmd I/O to complete in
order to make forward progress.
Note, there is a v4.1+ se_cmd->cmd_kref reference leak bug for
TMR ABORT_TASK during simultaneous target back-end I/O completion
timeouts here:
http://www.spinics.net/lists/target-devel/msg13530.html
If you are actively observing TMR ABORT_TASK preceding the hung task
timeout warnings above with v4.4.y + v4.2.y iser-target exports, then
it's likely the same bug. Please apply the patch on your v4.x setup to
verify.
If no TMR ABORT_TASK timeouts + session reinstatements are occurring on
your iser-target setup, then it is a separate bug.
^ permalink raw reply
* Re: [PATCH 03/28] [v2] infiniband: shut up a maybe-uninitialized warning
From: Haggai Eran @ 2016-10-18 6:47 UTC (permalink / raw)
To: Arnd Bergmann, Doug Ledford
Cc: Linus Torvalds, linux-kernel-u79uwXL29TY76Z2rM5mHXA, Sean Hefty,
Hal Rosenstock, Matan Barak, Leon Romanovsky, Sagi Grimberg,
Bart Van Assche, Alex Vesker, Guy Shapiro,
linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20161017220557.1688282-3-arnd-r2nGTMty4D4@public.gmane.org>
On 10/18/2016 1:05 AM, Arnd Bergmann wrote:
> @@ -1309,7 +1311,7 @@ static bool validate_net_dev(struct net_device *net_dev,
> static struct net_device *cma_get_net_dev(struct ib_cm_event *ib_event,
> const struct cma_req_info *req)
> {
> - struct sockaddr_storage listen_addr_storage, src_addr_storage;
> + struct sockaddr_storage listen_addr_storage = {}, src_addr_storage = {};
Doesn't this still translate to an extra initialization that Doug was
worried about?
Haggai
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 6/8] IB/hns: Replace counting semaphore event_sem with wait condition
From: Binoy Jayan @ 2016-10-18 5:16 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Doug Ledford, Sean Hefty, Hal Rosenstock,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux kernel mailing list
In-Reply-To: <5900385.6T4BAIyXjD@wuerfel>
On 18 October 2016 at 01:59, Arnd Bergmann <arnd-r2nGTMty4D4@public.gmane.org> wrote:
> On Monday, October 17, 2016 10:01:00 PM CEST Binoy Jayan wrote:
>> --- a/drivers/infiniband/hw/hns/hns_roce_cmd.c
>> +++ b/drivers/infiniband/hw/hns/hns_roce_cmd.c
>> @@ -248,10 +248,14 @@ static int hns_roce_cmd_mbox_wait(struct hns_roce_dev *hr_dev, u64 in_param,
>> {
>> int ret = 0;
>>
>> - down(&hr_dev->cmd.event_sem);
>> + wait_event(hr_dev->cmd.event_sem.wq,
>> + atomic_add_unless(&hr_dev->cmd.event_sem.count, -1, 0));
>> +
>> ret = __hns_roce_cmd_mbox_wait(hr_dev, in_param, out_param,
>> in_modifier, op_modifier, op, timeout);
>> - up(&hr_dev->cmd.event_sem);
>> +
>> + if (atomic_inc_return(&hr_dev->cmd.event_sem.count) == 1)
>> + wake_up(&hr_dev->cmd.event_sem.wq);
>>
>> return ret;
>> }
>
> This is the only interesting use of the event_sem that cares about
> the counting and it protects the __hns_roce_cmd_mbox_wait() from being
> entered too often. The count here is the number of size of the
> hr_dev->cmd.context[] array.
>
> However, that function already use a spinlock to protect that array
> and pick the correct context. I think changing the inner function
> to handle the case of 'no context available' by using a waitqueue
> without counting anything would be a reasonable transformation
> away from the semaphore.
>
> Arnd
Hi Arnd,
Thank you for replying for the questions. I''ll look for alternatives
for patches
6,7 and 8 and resend the series.
-Binoy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: building rdma-core from travis
From: Leon Romanovsky @ 2016-10-18 5:08 UTC (permalink / raw)
To: Hefty, Sean
Cc: linux-rdma (linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org)
In-Reply-To: <1828884A29C6694DAF28B7E6B8A82373AB096A91-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 780 bytes --]
On Tue, Oct 18, 2016 at 12:06:19AM +0000, Hefty, Sean wrote:
> > CMake Error at CMakeLists.txt:23 (cmake_minimum_required):
> >
> > CMake 2.8.11 or higher is required. You are running version 2.8.7
> >
> >
> > I can see that travis runs successfully against rdma-core. Was there a
> > magic setting used to work-around the travis version?
>
> It looks like rdma-core requires the 'trusty' build on travis...
Yes, you are right.
"# We need at least cmake 2.12, this means we need to use trusty."
http://marc.info/?l=linux-rdma&m=147587977913428&w=2
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox