* [Bug 178581] New: v4.9-rc1: Deadlock while unloading IPoIB
From: bugzilla-daemon-590EEB7GvNiWaY/ihj7yzEB+6BGkLq7r @ 2016-10-19 20:31 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
https://bugzilla.kernel.org/show_bug.cgi?id=178581
Bug ID: 178581
Summary: v4.9-rc1: Deadlock while unloading IPoIB
Product: Drivers
Version: 2.5
Kernel Version: v4.9-rc1
Hardware: x86-64
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Infiniband/RDMA
Assignee: drivers_infiniband-rdma-ztI5WcYan/vQLgFONoPN62D2FQJk+8+b@public.gmane.org
Reporter: bvanassche-HInyCGIudOg@public.gmane.org
Regression: No
Created attachment 241991
--> https://bugzilla.kernel.org/attachment.cgi?id=241991&action=edit
Lockdep output
The output produced by lockdep has been attached. Call trace from that
attachment:
Call Trace:
[<ffffffff81329695>] dump_stack+0x68/0x93
[<ffffffff810baafe>] print_circular_bug+0x1be/0x210
[<ffffffff810bd31c>] __lock_acquire+0x10fc/0x1270
[<ffffffff810bd8b9>] lock_acquire+0xe9/0x1d0
[<ffffffff8126275a>] __kernfs_remove+0x24a/0x310
[<ffffffff812632ae>] kernfs_remove+0x1e/0x30
[<ffffffff8126559c>] sysfs_remove_dir+0x4c/0x80
[<ffffffff8132bec8>] kobject_del+0x18/0x50
[<ffffffff81450fb0>] device_del+0x190/0x240
[<ffffffff81550a27>] netdev_unregister_kobject+0x57/0x60
[<ffffffff81527df0>] rollback_registered_many+0x220/0x370
[<ffffffff81527f6c>] rollback_registered+0x2c/0x40
[<ffffffff81528e54>] unregister_netdevice_queue+0x44/0x90
[<ffffffff81528eb7>] unregister_netdev+0x17/0x20
[<ffffffffa02bdc47>] ipoib_remove_one+0x97/0xe0 [ib_ipoib]
[<ffffffffa032c539>] ib_unregister_device+0xa9/0x150 [ib_core]
[<ffffffffa036d7e5>] mlx4_ib_remove+0x65/0x250 [mlx4_ib]
[<ffffffffa025131e>] mlx4_remove_device+0x6e/0x80 [mlx4_core]
[<ffffffffa0251738>] mlx4_unregister_device+0x48/0x90 [mlx4_core]
[<ffffffffa025364d>] mlx4_unload_one+0x7d/0x310 [mlx4_core]
[<ffffffffa02539cd>] mlx4_shutdown+0x5d/0x70 [mlx4_core]
[<ffffffff8136c768>] pci_device_shutdown+0x28/0x60
[<ffffffff81452f70>] device_shutdown+0xd0/0x1d0
[<ffffffff8108c7e1>] kernel_restart_prepare+0x31/0x40
[<ffffffff8108c88d>] kernel_restart+0xd/0x50
[<ffffffff8108cad9>] SYSC_reboot+0xf9/0x1e0
[<ffffffff8108cc09>] SyS_reboot+0x9/0x10
[<ffffffff8163ed2a>] entry_SYSCALL_64_fastpath+0x18/0xad
--
You are receiving this mail because:
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCHv12 0/3] rdmacg: IB/core: rdma controller support
From: Parav Pandit @ 2016-10-19 20:18 UTC (permalink / raw)
To: Tejun Heo
Cc: Leon Romanovsky, cgroups-u79uwXL29TY76Z2rM5mHXA, linux-rdma,
Li Zefan, Johannes Weiner, Doug Ledford, Christoph Hellwig,
Liran Liss, Hefty, Sean, Jason Gunthorpe, Haggai Eran,
james.l.morris-QHcLZuEGTsvQT0dZR+AlfA, Or Gerlitz, Matan Barak
In-Reply-To: <20161019200536.GC3044-piEFEHQLUPpN0TnZuCh8vA@public.gmane.org>
On Thu, Oct 20, 2016 at 1:35 AM, Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> Hello, Parav.
>
> On Thu, Oct 20, 2016 at 01:24:42AM +0530, Parav Pandit wrote:
>> userland can get the max numbers using other framework which is used
>> by control & data plane available in C library form or in form of
>> system tools.
>> I was preferring to get and set through same interface because,
>> It simplifies user land software which is often not written in C so
>> its likely that it needs to rely on system tools and parse the
>> content, iterate through devices etc.
>> Getting these info through rdma.max just makes it simple. There will
>> be logic built to read/write rdma.max in userland anyway, which can be
>> leveraged for percentage calculation instead of doing it from two
>> places.
>
> Yeah, I get that this can be convenient in this case but it isn't a
> generic approach. I'd much prefer keeping it in line with other
> resources.
>
Hmm. we don't have /proc/sys/kernel/pid_max type of simple interface
to get the max values for rdma resources.
rdma.max is close to that simplicity.
> Thanks.
>
> --
> tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: New providers in rdma-core
From: Jason Gunthorpe @ 2016-10-19 20:09 UTC (permalink / raw)
To: Adit Ranadive
Cc: Christoph Hellwig, Leon Romanovsky, Amrani, Ram, Lijun Ou,
Knut Omang, Doug Ledford, linux-rdma
In-Reply-To: <12243e3b-87cf-2bd4-d097-1dd0a7907f41-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
On Wed, Oct 19, 2016 at 12:51:02PM -0700, Adit Ranadive wrote:
> I see that rxe still has a rxe-abi header in the providers/rxe folder.
> Would that also move to the kernel as part of the conversion?
Ideally, but it is blocked by this:
> One of the things I'm unclear about what happends to the IBV structs
> included in the user-level ABI headers, for example, the ibv_get_context_resp
> here:
> https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx4/mlx4-abi.h#L48
This is the main thing preventing the use of the uapi headers - they
are actually *different* :(
It would be great if you have have some ideas on how to deal with
that.. This hasn't been tackled yet.
One choice is to write this:
struct mlx4_alloc_ucontext_resp_v3 {
struct ibv_get_context_resp ibv_resp;
__u32 qp_tab_size;
__u16 bf_reg_size;
__u16 bf_regs_per_page;
};
as:
struct user_mlx4_alloc_ucontext_resp_v3 {
struct ibv_get_context_resp ibv_resp;
struct mlx4_alloc_ucontext_resp_v3 udata;
};
Perhaps aided by a macro so we get high uniformity:
#define DECLARE_DRIVER_UDATA(struct_driver, struct_common)
struct user # struct_driver {
struct struct_common ibv_resp;
struct struct_driver udata;
};
Where mlx4_alloc_ucontext_resp_v3 comes from the uapi header, and is
not inlined, so each driver gets an 'abi wrapper' header that just
builds all these structs.
I'd probably advocate for that for new drivers if nothing more clever
comes forward. At least it is a step closer..
Another approach might be a different way to marshal the kernel call..
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCHv12 0/3] rdmacg: IB/core: rdma controller support
From: Tejun Heo @ 2016-10-19 20:05 UTC (permalink / raw)
To: Parav Pandit
Cc: Leon Romanovsky, cgroups-u79uwXL29TY76Z2rM5mHXA, linux-rdma,
Li Zefan, Johannes Weiner, Doug Ledford, Christoph Hellwig,
Liran Liss, Hefty, Sean, Jason Gunthorpe, Haggai Eran,
james.l.morris-QHcLZuEGTsvQT0dZR+AlfA, Or Gerlitz, Matan Barak
In-Reply-To: <CAG53R5X5dyo7J-UkeMxi_mSxgv=c54fV=anuCZtmf9kaYwDbPw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
Hello, Parav.
On Thu, Oct 20, 2016 at 01:24:42AM +0530, Parav Pandit wrote:
> userland can get the max numbers using other framework which is used
> by control & data plane available in C library form or in form of
> system tools.
> I was preferring to get and set through same interface because,
> It simplifies user land software which is often not written in C so
> its likely that it needs to rely on system tools and parse the
> content, iterate through devices etc.
> Getting these info through rdma.max just makes it simple. There will
> be logic built to read/write rdma.max in userland anyway, which can be
> leveraged for percentage calculation instead of doing it from two
> places.
Yeah, I get that this can be convenient in this case but it isn't a
generic approach. I'd much prefer keeping it in line with other
resources.
Thanks.
--
tejun
^ permalink raw reply
* Re: [PATCH 1/3] memremap.c : Add support for ZONE_DEVICE IO memory with struct pages.
From: Dan Williams @ 2016-10-19 20:01 UTC (permalink / raw)
To: Stephen Bates
Cc: Jason Gunthorpe, sbates-Rgftl6RXld5BDgjK7y7TUQ,
linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Jonathan Corbet,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
jim.macdonald-FgSLVYC75IpWk0Htik3J/w,
linux-block-u79uwXL29TY76Z2rM5mHXA, Linux MM, Jens Axboe,
Christoph Hellwig, haggaie-VPRAkNaXOzVWk0Htik3J/w
In-Reply-To: <20161019184028.GB16550-Fu/sLxIRQ7hz7qcAzjI/l7KmtST0zaCzZeezCHUQhQ4@public.gmane.org>
On Wed, Oct 19, 2016 at 11:40 AM, Stephen Bates <sbates-pv7U853sEMVWk0Htik3J/w@public.gmane.org> wrote:
> On Wed, Oct 19, 2016 at 10:50:25AM -0700, Dan Williams wrote:
>> On Tue, Oct 18, 2016 at 2:42 PM, Stephen Bates <sbates-pv7U853sEMVWk0Htik3J/w@public.gmane.org> wrote:
>> > From: Logan Gunthorpe <logang-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org>
>> >
>> > We build on recent work that adds memory regions owned by a device
>> > driver (ZONE_DEVICE) [1] and to add struct page support for these new
>> > regions of memory [2].
>> >
>> > 1. Add an extra flags argument into dev_memremap_pages to take in a
>> > MEMREMAP_XX argument. We update the existing calls to this function to
>> > reflect the change.
>> >
>> > 2. For completeness, we add MEMREMAP_WT support to the memremap;
>> > however we have no actual need for this functionality.
>> >
>> > 3. We add the static functions, add_zone_device_pages and
>> > remove_zone_device pages. These are similar to arch_add_memory except
>> > they don't create the memory mapping. We don't believe these need to be
>> > made arch specific, but are open to other opinions.
>> >
>> > 4. dev_memremap_pages and devm_memremap_pages_release are updated to
>> > treat IO memory slightly differently. For IO memory we use a combination
>> > of the appropriate io_remap function and the zone_device pages functions
>> > created above. A flags variable and kaddr pointer are added to struct
>> > page_mem to facilitate this for the release function. We also set up
>> > the page attribute tables for the mapped region correctly based on the
>> > desired mapping.
>> >
>>
>> This description says "what" is being done, but not "why".
>
> Hi Dan
>
> We discuss the motivation in the cover letter.
>
>>
>> In the cover letter, "[PATCH 0/3] iopmem : A block device for PCIe
>> memory", it mentions that the lack of I/O coherency is a known issue
>> and users of this functionality need to be cognizant of the pitfalls.
>> If that is the case why do we need support for different cpu mapping
>> types than the default write-back cache setting? It's up to the
>> application to handle cache cpu flushing similar to what we require of
>> device-dax users in the persistent memory case.
>
> Some of the iopmem hardware we have tested has certain alignment
> restrictions on BAR accesses. At the very least we require write
> combine mappings for these. We then felt it appropriate to add the
> other mappings for the sake of completeness.
If the device can support write-combine then it can support bursts, so
I wonder why it couldn't support read bursts for cache fills...
^ permalink raw reply
* Re: [PATCH 0/3] iopmem : A block device for PCIe memory
From: Dan Williams @ 2016-10-19 19:58 UTC (permalink / raw)
To: Stephen Bates
Cc: linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org,
linux-rdma, linux-block, Linux MM, Ross Zwisler, Matthew Wilcox,
Jason Gunthorpe, haggaie, Christoph Hellwig, Jens Axboe,
Jonathan Corbet, jim.macdonald, sbates, Logan Gunthorpe,
David Woodhouse, Raj, Ashok
In-Reply-To: <20161019184814.GC16550@cgy1-donard.priv.deltatee.com>
On Wed, Oct 19, 2016 at 11:48 AM, Stephen Bates <sbates@raithlin.com> wrote:
> On Tue, Oct 18, 2016 at 08:51:15PM -0700, Dan Williams wrote:
>> [ adding Ashok and David for potential iommu comments ]
>>
>
> Hi Dan
>
> Thanks for adding Ashok and David!
>
>>
>> I agree with the motivation and the need for a solution, but I have
>> some questions about this implementation.
>>
>> >
>> > Consumers
>> > ---------
>> >
>> > We provide a PCIe device driver in an accompanying patch that can be
>> > used to map any PCIe BAR into a DAX capable block device. For
>> > non-persistent BARs this simply serves as an alternative to using
>> > system memory bounce buffers. For persistent BARs this can serve as an
>> > additional storage device in the system.
>>
>> Why block devices? I wonder if iopmem was initially designed back
>> when we were considering enabling DAX for raw block devices. However,
>> that support has since been ripped out / abandoned. You currently
>> need a filesystem on top of a block-device to get DAX operation.
>> Putting xfs or ext4 on top of PCI-E memory mapped range seems awkward
>> if all you want is a way to map the bar for another PCI-E device in
>> the topology.
>>
>> If you're only using the block-device as a entry-point to create
>> dax-mappings then a device-dax (drivers/dax/) character-device might
>> be a better fit.
>>
>
> We chose a block device because we felt it was intuitive for users to
> carve up a memory region but putting a DAX filesystem on it and creating
> files on that DAX aware FS. It seemed like a convenient way to
> partition up the region and to be easily able to get the DMA address
> for the memory backing the device.
>
> That said I would be very keen to get other peoples thoughts on how
> they would like to see this done. And I know some people have had some
> reservations about using DAX mounted FS to do this in the past.
I guess it depends on the expected size of these devices BARs, but I
get the sense they may be smaller / more precious such that you
wouldn't want to spend capacity on filesystem metadata? For the target
use case is it assumed that these device BARs are always backed by
non-volatile memory? Otherwise this is a mkfs each boot for a
volatile device.
>>
>> > 2. Memory Segment Spacing. This patch has the same limitations that
>> > ZONE_DEVICE does in that memory regions must be spaces at least
>> > SECTION_SIZE bytes part. On x86 this is 128MB and there are cases where
>> > BARs can be placed closer together than this. Thus ZONE_DEVICE would not
>> > be usable on neighboring BARs. For our purposes, this is not an issue as
>> > we'd only be looking at enabling a single BAR in a given PCIe device.
>> > More exotic use cases may have problems with this.
>>
>> I'm working on patches for 4.10 to allow mixing multiple
>> devm_memremap_pages() allocations within the same physical section.
>> Hopefully this won't be a problem going forward.
>>
>
> Thanks Dan. Your patches will help address the problem of how to
> partition a /dev/dax device but they don't help the case then BARs
> themselves are small, closely spaced and non-segment aligned. However
> I think most people using iopmem will want to use reasonbly large
> BARs so I am not sure item 2 is that big of an issue.
I think you might have misunderstood what I'm proposing. The patches
I'm working on are separate from a facility to carve up a /dev/dax
device. The effort is to allow devm_memremap_pages() to maintain
several allocations within the same 128MB section. I need this for
persistent memory to handle platforms that mix pmem and system-ram in
the same section. I want to be able to map ZONE_DEVICE pages for a
portion of a section and be able to remove portions of section that
may collide with allocations of a different lifetime.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCHv12 0/3] rdmacg: IB/core: rdma controller support
From: Parav Pandit @ 2016-10-19 19:54 UTC (permalink / raw)
To: Tejun Heo
Cc: Leon Romanovsky, cgroups-u79uwXL29TY76Z2rM5mHXA, linux-rdma,
Li Zefan, Johannes Weiner, Doug Ledford, Christoph Hellwig,
Liran Liss, Hefty, Sean, Jason Gunthorpe, Haggai Eran,
james.l.morris-QHcLZuEGTsvQT0dZR+AlfA, Or Gerlitz, Matan Barak
In-Reply-To: <20161019192006.GB3044-piEFEHQLUPpN0TnZuCh8vA@public.gmane.org>
Hi Tejun,
On Thu, Oct 20, 2016 at 12:50 AM, Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> Hello,
>
> On Thu, Oct 20, 2016 at 12:33:53AM +0530, Parav Pandit wrote:
>> > or config changes in one of the ancestors? What "max" means is "no
>> > limit is imposed" which is different from "limit it to 100% of what's
>> > currently available".
>>
>> Charging is hierarchical for rdmacg too.
>> rdma.max configuration exist at all the levels so ancestors change
>> won't affect its children.
>> rdma.max absolute (or future percentage) value is with reference to
>> the total device resources.
>
> Ah, right, the percentage is out of the total device resources
> regardless of the hierarchical restrictions. I still don't think it's
> a good idea for rdmacg to deviate from the common interface
> conventions. If you want to do the percentage calculation in the
> userland and the base numbers are system-wide numbers which are
> hardware dependent, it'd be best if there's an existing place where
> the numbers can be exposed naturally. That'd be more in line with
> others too. The amount of total resources available for the device
> isn't tied to cgroup after all.
>
userland can get the max numbers using other framework which is used
by control & data plane available in C library form or in form of
system tools.
I was preferring to get and set through same interface because,
It simplifies user land software which is often not written in C so
its likely that it needs to rely on system tools and parse the
content, iterate through devices etc.
Getting these info through rdma.max just makes it simple. There will
be logic built to read/write rdma.max in userland anyway, which can be
leveraged for percentage calculation instead of doing it from two
places.
> Thanks.
>
> --
> tejun
^ permalink raw reply
* Re: New providers in rdma-core
From: Adit Ranadive @ 2016-10-19 19:51 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Christoph Hellwig, Leon Romanovsky, Amrani, Ram, Lijun Ou,
Knut Omang, Doug Ledford, linux-rdma
In-Reply-To: <20161019175926.GD29879-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
On Wed, Oct 19, 2016 at 11:59:26AM -0600, Jason Gunthorpe wrote:
> On Wed, Oct 19, 2016 at 10:28:36AM -0700, Adit Ranadive wrote:
>
> > > Requiring installed super updtodate kernel headers isn't going to
> > > help development. It would be good to keep a copy of the kernel
> > > headers in the rdma-core repo, similar to what most kernel tools
> > > tied to bleeding edge ABIs do.
> >
> > Thanks for the heads up. I had a few questions regarding the ABI stuff.
> > What happens in case of providers that arent part of the kernel as yet?
> > I would like to send out a pull request soonish for libpvrdma. I would
> > need to include the abi files in that request since cmake would fail.
>
> Follow the pattern rxe is using:
>
> CHECK_INCLUDE_FILE("rdma/rdma_user_rxe.h" HAVE_RDMA_USER_RXE)
> RDMA_DoFixup("${HAVE_RDMA_USER_RXE}" "rdma/rdma_user_rxe.h")
>
> Which uses the compat header file in
> buildlib/fixup-include/rdma-rdma_user_rxe.h, which must be a
> byte-for-byte copy of the kernel header.
I see that rxe still has a rxe-abi header in the providers/rxe folder.
Would that also move to the kernel as part of the conversion?
One of the things I'm unclear about what happends to the IBV structs
included in the user-level ABI headers, for example, the ibv_get_context_resp
here:
https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx4/mlx4-abi.h#L48
Since these are not present in the kernel, how would the libibverbs/core
uverbs stack handle this? Would there be forthcoming changes for that too?
> > Or do you want me to wait till Doug adds the PVRDMA driver to the tree?
>
> FWIW, we can't take a rdma-core provider until the kernel side is
> accepted, but you can certainly start the process now and get it to an
> OK state. Doug will be able to take both parts simultaneously.
>
Makes sense.
Thanks,
Adit
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCHv12 0/3] rdmacg: IB/core: rdma controller support
From: Tejun Heo @ 2016-10-19 19:20 UTC (permalink / raw)
To: Parav Pandit
Cc: Leon Romanovsky, cgroups-u79uwXL29TY76Z2rM5mHXA, linux-rdma,
Li Zefan, Johannes Weiner, Doug Ledford, Christoph Hellwig,
Liran Liss, Hefty, Sean, Jason Gunthorpe, Haggai Eran,
james.l.morris-QHcLZuEGTsvQT0dZR+AlfA, Or Gerlitz, Matan Barak
In-Reply-To: <CAG53R5WUyA7JBn=PeivUc5F5k210xf_HccPXFt3r7ZGYHOPaGA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
Hello,
On Thu, Oct 20, 2016 at 12:33:53AM +0530, Parav Pandit wrote:
> > or config changes in one of the ancestors? What "max" means is "no
> > limit is imposed" which is different from "limit it to 100% of what's
> > currently available".
>
> Charging is hierarchical for rdmacg too.
> rdma.max configuration exist at all the levels so ancestors change
> won't affect its children.
> rdma.max absolute (or future percentage) value is with reference to
> the total device resources.
Ah, right, the percentage is out of the total device resources
regardless of the hierarchical restrictions. I still don't think it's
a good idea for rdmacg to deviate from the common interface
conventions. If you want to do the percentage calculation in the
userland and the base numbers are system-wide numbers which are
hardware dependent, it'd be best if there's an existing place where
the numbers can be exposed naturally. That'd be more in line with
others too. The amount of total resources available for the device
isn't tied to cgroup after all.
Thanks.
--
tejun
^ permalink raw reply
* Re: [PATCHv12 0/3] rdmacg: IB/core: rdma controller support
From: Parav Pandit @ 2016-10-19 19:03 UTC (permalink / raw)
To: Tejun Heo
Cc: Leon Romanovsky, cgroups-u79uwXL29TY76Z2rM5mHXA, linux-rdma,
Li Zefan, Johannes Weiner, Doug Ledford, Christoph Hellwig,
Liran Liss, Hefty, Sean, Jason Gunthorpe, Haggai Eran,
james.l.morris-QHcLZuEGTsvQT0dZR+AlfA, Or Gerlitz, Matan Barak
In-Reply-To: <20161019143345.GA18532-piEFEHQLUPpN0TnZuCh8vA@public.gmane.org>
Hi Tejun,
On Wed, Oct 19, 2016 at 8:03 PM, Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> Hello,
>
> On Wed, Oct 19, 2016 at 03:04:52PM +0530, Parav Pandit wrote:
>> > Heh, we can go for per-mil and use %0 as the suffix if absolutely
>> > necessary but is this a real issue?
>>
>> We need to provide current version to exceed the 100 containers limit first. :-)
>> Please let me know if such user interface already exist as reference point.
>> I prefer to avoid introducing such configuration.
>
> We don't have such usage yet. rdmacg would be the first one doing
> something like this.
>
>> > I think it's a lot more consistent to implement all absoulte limits
>> > through max.
>>
>> I agree. I will implement reporting actual max values instead of
>> reporting "max" string so that percentage configuration can be done by
>> the user space tools.
>> (post merging of v12).
>> Waiting for direction from Doug.
>
> That doesn't sound like a good idea. What happens when the amount of
> available resources changes either through underlying resource changes
To my knowledge device configuration cannot change on the fly while
its still active.
PCIe SR-IOV VF might be able to do so. In that case we might need to
introduce event notifier framework.
This can be possibility regardless of percentage configuration if such
device comes up.
However this appears far fetched.
> or config changes in one of the ancestors? What "max" means is "no
> limit is imposed" which is different from "limit it to 100% of what's
> currently available".
Charging is hierarchical for rdmacg too.
rdma.max configuration exist at all the levels so ancestors change
won't affect its children.
rdma.max absolute (or future percentage) value is with reference to
the total device resources.
> Thanks.
>
> --
> tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH rdma-core 2/4] glue/redhat: add udev/systemd/etc infrastructure bits
From: Jason Gunthorpe @ 2016-10-19 18:59 UTC (permalink / raw)
To: Weiny, Ira
Cc: Jarod Wilson, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Doug Ledford, Hefty, Sean
In-Reply-To: <2807E5FD2F6FDA4886F6618EAC48510E24F0DF57-8k97q/ur5Z2krb+BlOpmy7fspsVTdybXVpNB7YpNyf8@public.gmane.org>
On Wed, Oct 19, 2016 at 06:38:52PM +0000, Weiny, Ira wrote:
> I have the C code ported to rdma-core but how do I do this in cmake?
Attached, you will also need
target_link_libraries(rdma-ndd ${UDEV_LIBRARIES})
> PKG_CHECK_EXISTS(libudev >= 218, [with_dev_logging=no],
> [with_udev_logging=yes])
> if test "$with_udev_logging" = "yes"; then
> AC_DEFINE_UNQUOTED([HAVE_UDEV_LOGGING], 1,
> [whether libudev logging can be used])
Dump HAVE_UDEV_LOGGING from the code, upstream deleted support for it:
https://www.redhat.com/archives/libvir-list/2014-December/msg00749.html
src.c:3:2: warning: 'udev_set_log_fn' is deprecated (declared at /usr/include/libudev.h:41) [-Wdeprecated-declarations]
int main(int argc,const char *argv[]) {udev_set_log_fn(NULL, NULL); return 0;}
^
> I've found a "modules" file which looks like it has a compatible BSD
> license and could be added but is there a better way?
Basically the right idea, but I prefer to avoid pkgconfig whenever
possible as it screws up cross compiling.
> I also have to convert the man page from *.rst to man in some way...
> Would it be ok if I put a dependency on rst2man in the repo?
We have talked about it, maybe for now just include both the .rst and
the rst2man output and we can revisit it..
Jason
>From b0062acc238e06289aa946dacd5c534cf6c68d7d Mon Sep 17 00:00:00 2001
From: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
Date: Wed, 19 Oct 2016 12:57:25 -0600
Subject: [PATCH] Add a dependency on libudev
incomplete, needs the RH stuff too.
Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
.travis.yml | 1 +
CMakeLists.txt | 4 ++++
README.md | 2 +-
buildlib/FindUDev.cmake | 10 ++++++++++
debian/control | 1 +
5 files changed, 17 insertions(+), 1 deletion(-)
create mode 100644 buildlib/FindUDev.cmake
diff --git a/.travis.yml b/.travis.yml
index d81b699294eb..d9c36cc9c649 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -24,6 +24,7 @@ addons:
- gcc-6
- libnl-3-dev
- libnl-route-3-dev
+ - libudev-dev
- make
- ninja-build
- pkg-config
diff --git a/CMakeLists.txt b/CMakeLists.txt
index a23aa860e6d3..9402bacf70ce 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -237,6 +237,10 @@ if (NOT NL_KIND EQUAL 0)
set(CMAKE_REQUIRED_INCLUDES "${SAFE_CMAKE_REQUIRED_INCLUDES}")
endif()
+# udev
+find_package(UDev REQUIRED)
+include_directories(${UDEV_INCLUDE_DIRS})
+
# Statically determine sizeof(long), this is largely unnecessary, no new code
# should rely on this.
check_type_size("long" SIZEOF_LONG BUILTIN_TYPES_ONLY LANGUAGE C)
diff --git a/README.md b/README.md
index 66aee3f49f00..d24fd0bf2606 100644
--- a/README.md
+++ b/README.md
@@ -48,7 +48,7 @@ only load from the system path.
### Debian Derived
```sh
-$ apt-get install build-essential cmake gcc libnl-3-dev libnl-route-3-dev ninja-build pkg-config valgrind
+$ apt-get install build-essential cmake gcc libudev-dev libnl-3-dev libnl-route-3-dev ninja-build pkg-config valgrind
```
### Fedora
diff --git a/buildlib/FindUDev.cmake b/buildlib/FindUDev.cmake
new file mode 100644
index 000000000000..ce05ddf991a1
--- /dev/null
+++ b/buildlib/FindUDev.cmake
@@ -0,0 +1,10 @@
+# COPYRIGHT (c) 2016 Obsidian Research Corporation. See COPYING file
+
+find_library(LIBUDEV_LIBRARY NAMES udev libudev)
+
+set(UDEV_LIBRARIES ${LIBUDEV_LIBRARY})
+
+include(FindPackageHandleStandardArgs)
+find_package_handle_standard_args(UDev REQUIRED_VARS LIBUDEV_LIBRARY)
+
+mark_as_advanced(LIBUDEV_LIBRARY)
diff --git a/debian/control b/debian/control
index 2335d1f4814d..ed9850a348be 100644
--- a/debian/control
+++ b/debian/control
@@ -8,6 +8,7 @@ Build-Depends: build-essential,
dh-systemd,
dpkg-dev (>= 1.17),
gcc,
+ libudev-dev,
libnl-3-dev,
libnl-route-3-dev,
make,
--
2.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH 0/3] iopmem : A block device for PCIe memory
From: Stephen Bates @ 2016-10-19 18:48 UTC (permalink / raw)
To: Dan Williams
Cc: linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
linux-block-u79uwXL29TY76Z2rM5mHXA, Linux MM, Ross Zwisler,
Matthew Wilcox, jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/,
haggaie-VPRAkNaXOzVWk0Htik3J/w, Christoph Hellwig, Jens Axboe,
Jonathan Corbet, jim.macdonald-FgSLVYC75IpWk0Htik3J/w,
sbates-Rgftl6RXld5BDgjK7y7TUQ, Logan Gunthorpe, David Woodhouse,
Raj, Ashok
In-Reply-To: <CAPcyv4gJ_c-6s2BUjsu6okR1EF53R+KNuXnOc5jv0fuwJaa3cQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Tue, Oct 18, 2016 at 08:51:15PM -0700, Dan Williams wrote:
> [ adding Ashok and David for potential iommu comments ]
>
Hi Dan
Thanks for adding Ashok and David!
>
> I agree with the motivation and the need for a solution, but I have
> some questions about this implementation.
>
> >
> > Consumers
> > ---------
> >
> > We provide a PCIe device driver in an accompanying patch that can be
> > used to map any PCIe BAR into a DAX capable block device. For
> > non-persistent BARs this simply serves as an alternative to using
> > system memory bounce buffers. For persistent BARs this can serve as an
> > additional storage device in the system.
>
> Why block devices? I wonder if iopmem was initially designed back
> when we were considering enabling DAX for raw block devices. However,
> that support has since been ripped out / abandoned. You currently
> need a filesystem on top of a block-device to get DAX operation.
> Putting xfs or ext4 on top of PCI-E memory mapped range seems awkward
> if all you want is a way to map the bar for another PCI-E device in
> the topology.
>
> If you're only using the block-device as a entry-point to create
> dax-mappings then a device-dax (drivers/dax/) character-device might
> be a better fit.
>
We chose a block device because we felt it was intuitive for users to
carve up a memory region but putting a DAX filesystem on it and creating
files on that DAX aware FS. It seemed like a convenient way to
partition up the region and to be easily able to get the DMA address
for the memory backing the device.
That said I would be very keen to get other peoples thoughts on how
they would like to see this done. And I know some people have had some
reservations about using DAX mounted FS to do this in the past.
>
> > 2. Memory Segment Spacing. This patch has the same limitations that
> > ZONE_DEVICE does in that memory regions must be spaces at least
> > SECTION_SIZE bytes part. On x86 this is 128MB and there are cases where
> > BARs can be placed closer together than this. Thus ZONE_DEVICE would not
> > be usable on neighboring BARs. For our purposes, this is not an issue as
> > we'd only be looking at enabling a single BAR in a given PCIe device.
> > More exotic use cases may have problems with this.
>
> I'm working on patches for 4.10 to allow mixing multiple
> devm_memremap_pages() allocations within the same physical section.
> Hopefully this won't be a problem going forward.
>
Thanks Dan. Your patches will help address the problem of how to
partition a /dev/dax device but they don't help the case then BARs
themselves are small, closely spaced and non-segment aligned. However
I think most people using iopmem will want to use reasonbly large
BARs so I am not sure item 2 is that big of an issue.
> I haven't yet grokked the motivation for this, but I'll go comment on
> that separately.
Thanks Dan!
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 1/3] memremap.c : Add support for ZONE_DEVICE IO memory with struct pages.
From: Stephen Bates @ 2016-10-19 18:40 UTC (permalink / raw)
To: Dan Williams
Cc: linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org,
linux-rdma, linux-block, Linux MM, Ross Zwisler, Matthew Wilcox,
Jason Gunthorpe, haggaie, Christoph Hellwig, Jens Axboe,
Jonathan Corbet, jim.macdonald, sbates, Logan Gunthorpe
In-Reply-To: <CAPcyv4gmiqMNb+Q88Mf-9fFb4z4uAfWbbEWrv42OBH8838SSPQ@mail.gmail.com>
On Wed, Oct 19, 2016 at 10:50:25AM -0700, Dan Williams wrote:
> On Tue, Oct 18, 2016 at 2:42 PM, Stephen Bates <sbates@raithlin.com> wrote:
> > From: Logan Gunthorpe <logang@deltatee.com>
> >
> > We build on recent work that adds memory regions owned by a device
> > driver (ZONE_DEVICE) [1] and to add struct page support for these new
> > regions of memory [2].
> >
> > 1. Add an extra flags argument into dev_memremap_pages to take in a
> > MEMREMAP_XX argument. We update the existing calls to this function to
> > reflect the change.
> >
> > 2. For completeness, we add MEMREMAP_WT support to the memremap;
> > however we have no actual need for this functionality.
> >
> > 3. We add the static functions, add_zone_device_pages and
> > remove_zone_device pages. These are similar to arch_add_memory except
> > they don't create the memory mapping. We don't believe these need to be
> > made arch specific, but are open to other opinions.
> >
> > 4. dev_memremap_pages and devm_memremap_pages_release are updated to
> > treat IO memory slightly differently. For IO memory we use a combination
> > of the appropriate io_remap function and the zone_device pages functions
> > created above. A flags variable and kaddr pointer are added to struct
> > page_mem to facilitate this for the release function. We also set up
> > the page attribute tables for the mapped region correctly based on the
> > desired mapping.
> >
>
> This description says "what" is being done, but not "why".
Hi Dan
We discuss the motivation in the cover letter.
>
> In the cover letter, "[PATCH 0/3] iopmem : A block device for PCIe
> memory", it mentions that the lack of I/O coherency is a known issue
> and users of this functionality need to be cognizant of the pitfalls.
> If that is the case why do we need support for different cpu mapping
> types than the default write-back cache setting? It's up to the
> application to handle cache cpu flushing similar to what we require of
> device-dax users in the persistent memory case.
Some of the iopmem hardware we have tested has certain alignment
restrictions on BAR accesses. At the very least we require write
combine mappings for these. We then felt it appropriate to add the
other mappings for the sake of completeness.
Cheers
Stephen
^ permalink raw reply
* RE: [PATCH rdma-core 2/4] glue/redhat: add udev/systemd/etc infrastructure bits
From: Weiny, Ira @ 2016-10-19 18:38 UTC (permalink / raw)
To: Jarod Wilson, Jason Gunthorpe
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Doug Ledford,
Hefty, Sean
In-Reply-To: <20161018145104.GT14983-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Ok,
I have the C code ported to rdma-core but how do I do this in cmake?
AS_IF([test x$rdmandd = xyes], [
PKG_CHECK_MODULES([UDEV], [libudev])
AC_CONFIG_FILES([doc/man/rdma-ndd.8 \
etc/rdma-ndd.init \
etc/rdma-ndd.service])
AC_SUBST([UDEV_CFLAGS])
AC_SUBST([UDEV_LIBS])
if test "$with_udev" = "yes"; then
PKG_CHECK_EXISTS(libudev >= 218, [with_dev_logging=no],
[with_udev_logging=yes])
if test "$with_udev_logging" = "yes"; then
AC_DEFINE_UNQUOTED([HAVE_UDEV_LOGGING], 1,
[whether libudev logging can be used])
fi
fi
])
I've found a "modules" file which looks like it has a compatible BSD license and could be added but is there a better way?
http://fossies.org/linux/flightgear/CMakeModules/FindUDev.cmake
I also have to convert the man page from *.rst to man in some way... Would it be ok if I put a dependency on rst2man in the repo?
Ira
> -----Original Message-----
> From: Jarod Wilson [mailto:jarod-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
> Sent: Tuesday, October 18, 2016 7:51 AM
> To: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
> Cc: Weiny, Ira <ira.weiny-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Doug
> Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>; Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Subject: Re: [PATCH rdma-core 2/4] glue/redhat: add udev/systemd/etc
> infrastructure bits
>
> On Mon, Oct 17, 2016 at 02:13:09PM -0600, Jason Gunthorpe wrote:
> > On Mon, Oct 17, 2016 at 07:10:46PM +0000, Weiny, Ira wrote:
> >
> > > What I have been worried about is conflicts between infiniband-diags
> > > and the new rdma-core. RH made a separate package out of rdma-ndd
> > > so that would be easy but I don't think other distros have.
> >
> > > So how do you obsolete "part" of a package?
> >
> > The distros know how to do this, they just conflict with the old
> > version of infiniband-diags.
>
> Yeah, basically, you spin up a new infiniband-diags package that doesn't have
> rdma-ndd and a new rdma-core package that includes it, along with (in
> rpm-ese) 'Conflicts: infiniband-diags < <first version w/o rdma-ndd>', and that
> should pretty much cover it.
>
> --
> Jarod Wilson
> jarod-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 03/12] IB/hfi1: Fix an Oops on pci device force remove
From: Tadeusz Struk @ 2016-10-19 18:01 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Dennis Dalessandro, dledford-H+wXaHxf7aLQT0dZR+AlfA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Dean Luick, Ira Weiny
In-Reply-To: <20161017160731.GA5679-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
Hi Jason,
On 10/17/2016 09:07 AM, Jason Gunthorpe wrote:
>> + atomic_inc(&dd->user_refcount);
> This doesn't look like enough locking on acquire.. What prevents the
> driver from already having been unloaded at this point?
Thanks for your feedback. We will address this in v2.
>
> Shouldn't this be a while loop to avoid races?
>
> .. and shouldn't the inc 'latch off' once the removal process starts?
>
> I think you need to study how kernfs_drain is implemented and
> basically copy that approach, including the locking.
as to kernfs_drain approach, I think the same mechanism as in ib_verbs
will work better here.
>
> FWIW, it is much easier and probably long term better to arrange
> things so the FD can remain open but just return ENODEV to all calls.
> Eg the srcu based approach in uverbs.
We need to hold the device because not all the clients will talk to the
device via the FD. The PIO clients for instance don't go to kernel.
Thanks,
--
TS
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: New providers in rdma-core
From: Jason Gunthorpe @ 2016-10-19 17:59 UTC (permalink / raw)
To: Adit Ranadive
Cc: Christoph Hellwig, Leon Romanovsky, Amrani, Ram, Lijun Ou,
Knut Omang, Doug Ledford, linux-rdma
In-Reply-To: <6b57d79c-dbd3-cd9b-061e-27a3d3968845-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
On Wed, Oct 19, 2016 at 10:28:36AM -0700, Adit Ranadive wrote:
> > Requiring installed super updtodate kernel headers isn't going to
> > help development. It would be good to keep a copy of the kernel
> > headers in the rdma-core repo, similar to what most kernel tools
> > tied to bleeding edge ABIs do.
>
> Thanks for the heads up. I had a few questions regarding the ABI stuff.
> What happens in case of providers that arent part of the kernel as yet?
> I would like to send out a pull request soonish for libpvrdma. I would
> need to include the abi files in that request since cmake would fail.
Follow the pattern rxe is using:
CHECK_INCLUDE_FILE("rdma/rdma_user_rxe.h" HAVE_RDMA_USER_RXE)
RDMA_DoFixup("${HAVE_RDMA_USER_RXE}" "rdma/rdma_user_rxe.h")
Which uses the compat header file in
buildlib/fixup-include/rdma-rdma_user_rxe.h, which must be a
byte-for-byte copy of the kernel header.
> Or do you want me to wait till Doug adds the PVRDMA driver to the tree?
FWIW, we can't take a rdma-core provider until the kernel side is
accepted, but you can certainly start the process now and get it to an
OK state. Doug will be able to take both parts simultaneously.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 1/3] memremap.c : Add support for ZONE_DEVICE IO memory with struct pages.
From: Dan Williams @ 2016-10-19 17:50 UTC (permalink / raw)
To: Stephen Bates
Cc: Jason Gunthorpe, sbates-Rgftl6RXld5BDgjK7y7TUQ,
linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Jonathan Corbet,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
jim.macdonald-FgSLVYC75IpWk0Htik3J/w,
linux-block-u79uwXL29TY76Z2rM5mHXA, Linux MM, Jens Axboe,
Christoph Hellwig, haggaie-VPRAkNaXOzVWk0Htik3J/w
In-Reply-To: <1476826937-20665-2-git-send-email-sbates-pv7U853sEMVWk0Htik3J/w@public.gmane.org>
On Tue, Oct 18, 2016 at 2:42 PM, Stephen Bates <sbates-pv7U853sEMVWk0Htik3J/w@public.gmane.org> wrote:
> From: Logan Gunthorpe <logang-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org>
>
> We build on recent work that adds memory regions owned by a device
> driver (ZONE_DEVICE) [1] and to add struct page support for these new
> regions of memory [2].
>
> 1. Add an extra flags argument into dev_memremap_pages to take in a
> MEMREMAP_XX argument. We update the existing calls to this function to
> reflect the change.
>
> 2. For completeness, we add MEMREMAP_WT support to the memremap;
> however we have no actual need for this functionality.
>
> 3. We add the static functions, add_zone_device_pages and
> remove_zone_device pages. These are similar to arch_add_memory except
> they don't create the memory mapping. We don't believe these need to be
> made arch specific, but are open to other opinions.
>
> 4. dev_memremap_pages and devm_memremap_pages_release are updated to
> treat IO memory slightly differently. For IO memory we use a combination
> of the appropriate io_remap function and the zone_device pages functions
> created above. A flags variable and kaddr pointer are added to struct
> page_mem to facilitate this for the release function. We also set up
> the page attribute tables for the mapped region correctly based on the
> desired mapping.
>
This description says "what" is being done, but not "why".
In the cover letter, "[PATCH 0/3] iopmem : A block device for PCIe
memory", it mentions that the lack of I/O coherency is a known issue
and users of this functionality need to be cognizant of the pitfalls.
If that is the case why do we need support for different cpu mapping
types than the default write-back cache setting? It's up to the
application to handle cache cpu flushing similar to what we require of
device-dax users in the persistent memory case.
^ permalink raw reply
* Re: New providers in rdma-core
From: Adit Ranadive @ 2016-10-19 17:28 UTC (permalink / raw)
To: Christoph Hellwig, Leon Romanovsky
Cc: Amrani, Ram, Lijun Ou, Knut Omang, Doug Ledford, Jason Gunthorpe,
linux-rdma
In-Reply-To: <20161019163913.GA30843-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
Hi Leon/Christoph,
On Wed, Oct 19, 2016 at 09:39:13AM -0700, Christoph Hellwig wrote:
> On Wed, Oct 19, 2016 at 07:18:04PM +0300, Leon Romanovsky wrote:
> > Hi All,
> >
> > I want to remind you that latest kernel is exposing different vendor
> > specific structures (include/uapi/rdma/*-abi.,h) and new coming user
> > space drivers should use these headers directly and not copy/paste the
> > code from them.
> >
> > In the future, we will convert already accepted drivers too.
>
> Requiring installed super updtodate kernel headers isn't going to
> help development. It would be good to keep a copy of the kernel
> headers in the rdma-core repo, similar to what most kernel tools
> tied to bleeding edge ABIs do.
Thanks for the heads up. I had a few questions regarding the ABI stuff.
What happens in case of providers that arent part of the kernel as yet?
I would like to send out a pull request soonish for libpvrdma. I would
need to include the abi files in that request since cmake would fail.
Or do you want me to wait till Doug adds the PVRDMA driver to the tree?
Thanks,
Adit
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Introduction of libqedr to the Consolidated Userspace RDMA Library Repo
From: Jason Gunthorpe @ 2016-10-19 17:24 UTC (permalink / raw)
To: Amrani, Ram
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Elior, Ariel,
Kalderon, Michal, Borundia, Rajesh,
dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <SN1PR07MB220719CE0C668A321520311DF8D20-mikhvbZlbf8TSoR2DauN2+FPX92sqiQdvxpqHgZTriW3zl9H0oFU5g@public.gmane.org>
On Wed, Oct 19, 2016 at 04:08:12PM +0000, Amrani, Ram wrote:
> Hi Jason,
> I've created a user account on github, forked rdma_plumbing and added libqedr in multiple patches,
> after I've verified that it builds warning free and in-line with previous cleanups.
> I've sent a pull request to linux-rdma rather than rdma-plumbing. Is that OK?
So far so good.
As Leon said, post your patches to the mailing list.
You also need to make sure it builds, Travis says the 32 bit builds
are no good (look at the pull request and click on the red X)
The github process from here is to make changes and then update your
branch on your github, that will reflect in the pull request. Eg you
can immediately fix the 32 bit issues and see that travis goes green.
I left some minor notes for you on github, the build system stuff
looks fine to me, and I didn't notice anything too unusual in a casual
browse. Didn't check if the code was any good..
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 00/10] mm: adjust get_user_pages* functions to explicitly pass FOLL_* flags
From: Dave Hansen @ 2016-10-19 17:23 UTC (permalink / raw)
To: Michal Hocko
Cc: linux-mips, linux-fbdev, Jan Kara, kvm, linux-sh, dri-devel,
linux-mm, netdev, sparclinux, linux-ia64, linux-s390,
linux-samsung-soc, linux-scsi, linux-rdma, x86, Hugh Dickins,
linux-media, Rik van Riel, intel-gfx, adi-buildroot-devel,
ceph-devel, linux-arm-kernel, Lorenzo Stoakes, linux-cris-kernel,
Linus Torvalds, linuxppc-dev, linux-kernel, linux-security-module,
linux-alpha, linux-fs
In-Reply-To: <20161019170127.GN24393@dhcp22.suse.cz>
On 10/19/2016 10:01 AM, Michal Hocko wrote:
> The question I had earlier was whether this has to be an explicit FOLL
> flag used by g-u-p users or we can just use it internally when mm !=
> current->mm
The reason I chose not to do that was that deferred work gets run under
a basically random 'current'. If we just use 'mm != current->mm', then
the deferred work will sometimes have pkeys enforced and sometimes not,
basically randomly.
We want to be consistent with whether they are enforced or not, so we
explicitly indicate that by calling the remote variant vs. plain.
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply
* Re: New providers in rdma-core
From: Jason Gunthorpe @ 2016-10-19 17:13 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Leon Romanovsky, Amrani, Ram, Lijun Ou,
aditr-pghWNbHTmq7QT0dZR+AlfA, Knut Omang, Doug Ledford,
linux-rdma
In-Reply-To: <20161019163913.GA30843-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
On Wed, Oct 19, 2016 at 09:39:13AM -0700, Christoph Hellwig wrote:
> Requiring installed super updtodate kernel headers isn't going to
> help development. It would be good to keep a copy of the kernel
> headers in the rdma-core repo, similar to what most kernel tools
> tied to bleeding edge ABIs do.
Yes, this is already happening, eg rdma/rdma_user_rxe.h is cached in
the tree, and I have a patch to move the rdma_netlink.h to that scheme
as well.
cmake has a scheme where it detects if the system header is too
old/missing and if so transparently arranges things so that
#include <rdma/rdma_user_rxe.h>
Uses the internal copy.
I also want to add a cmake option to use the kernel headers from a
kernel tree - nobody has been testing that changes to the kernel
headers work with the user space either, and eg the changes the
netlink broke stuff :/
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] IB/hns: Move HNS RoCE user vendor structures
From: Leon Romanovsky @ 2016-10-19 17:13 UTC (permalink / raw)
To: dledford-H+wXaHxf7aLQT0dZR+AlfA, oulijun-hv44wF8Li93QT0dZR+AlfA,
xavier.huwei-hv44wF8Li93QT0dZR+AlfA
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
This patch moves HNS vendor's specific structures to
common UAPI folder which will be visible to all consumers.
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
drivers/infiniband/hw/hns/hns_roce_cq.c | 2 +-
drivers/infiniband/hw/hns/hns_roce_main.c | 2 +-
drivers/infiniband/hw/hns/hns_roce_qp.c | 2 +-
include/uapi/rdma/Kbuild | 1 +
.../hw/hns/hns_roce_user.h => include/uapi/rdma/hns-abi.h | 9 +++++----
5 files changed, 9 insertions(+), 7 deletions(-)
rename drivers/infiniband/hw/hns/hns_roce_user.h => include/uapi/rdma/hns-abi.h (94%)
diff --git a/drivers/infiniband/hw/hns/hns_roce_cq.c b/drivers/infiniband/hw/hns/hns_roce_cq.c
index 0973659..d8a1764 100644
--- a/drivers/infiniband/hw/hns/hns_roce_cq.c
+++ b/drivers/infiniband/hw/hns/hns_roce_cq.c
@@ -35,7 +35,7 @@
#include "hns_roce_device.h"
#include "hns_roce_cmd.h"
#include "hns_roce_hem.h"
-#include "hns_roce_user.h"
+#include <rdma/hns-abi.h>
#include "hns_roce_common.h"
static void hns_roce_ib_cq_comp(struct hns_roce_cq *hr_cq)
diff --git a/drivers/infiniband/hw/hns/hns_roce_main.c b/drivers/infiniband/hw/hns/hns_roce_main.c
index 764e35a..354f100 100644
--- a/drivers/infiniband/hw/hns/hns_roce_main.c
+++ b/drivers/infiniband/hw/hns/hns_roce_main.c
@@ -37,7 +37,7 @@
#include <rdma/ib_user_verbs.h>
#include "hns_roce_common.h"
#include "hns_roce_device.h"
-#include "hns_roce_user.h"
+#include <rdma/hns-abi.h>
#include "hns_roce_hem.h"
/**
diff --git a/drivers/infiniband/hw/hns/hns_roce_qp.c b/drivers/infiniband/hw/hns/hns_roce_qp.c
index e86dd8d..5d13b6b 100644
--- a/drivers/infiniband/hw/hns/hns_roce_qp.c
+++ b/drivers/infiniband/hw/hns/hns_roce_qp.c
@@ -37,7 +37,7 @@
#include "hns_roce_common.h"
#include "hns_roce_device.h"
#include "hns_roce_hem.h"
-#include "hns_roce_user.h"
+#include <rdma/hns-abi.h>
#define SQP_NUM (2 * HNS_ROCE_MAX_PORTS)
diff --git a/include/uapi/rdma/Kbuild b/include/uapi/rdma/Kbuild
index f14ab7f..b54f10d 100644
--- a/include/uapi/rdma/Kbuild
+++ b/include/uapi/rdma/Kbuild
@@ -14,3 +14,4 @@ header-y += mlx5-abi.h
header-y += mthca-abi.h
header-y += nes-abi.h
header-y += ocrdma-abi.h
+header-y += hns-abi.h
diff --git a/drivers/infiniband/hw/hns/hns_roce_user.h b/include/uapi/rdma/hns-abi.h
similarity index 94%
rename from drivers/infiniband/hw/hns/hns_roce_user.h
rename to include/uapi/rdma/hns-abi.h
index a28f761..5d74019 100644
--- a/drivers/infiniband/hw/hns/hns_roce_user.h
+++ b/include/uapi/rdma/hns-abi.h
@@ -30,8 +30,10 @@
* SOFTWARE.
*/
-#ifndef _HNS_ROCE_USER_H
-#define _HNS_ROCE_USER_H
+#ifndef HNS_ABI_USER_H
+#define HNS_ABI_USER_H
+
+#include <linux/types.h>
struct hns_roce_ib_create_cq {
__u64 buf_addr;
@@ -49,5 +51,4 @@ struct hns_roce_ib_create_qp {
struct hns_roce_ib_alloc_ucontext_resp {
__u32 qp_tab_size;
};
-
-#endif /*_HNS_ROCE_USER_H */
+#endif /* HNS_ABI_USER_H */
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH 00/10] mm: adjust get_user_pages* functions to explicitly pass FOLL_* flags
From: Michal Hocko @ 2016-10-19 17:01 UTC (permalink / raw)
To: Dave Hansen
Cc: Lorenzo Stoakes, linux-mm, Linus Torvalds, Jan Kara, Hugh Dickins,
Rik van Riel, Mel Gorman, Andrew Morton, adi-buildroot-devel,
ceph-devel, dri-devel, intel-gfx, kvm, linux-alpha,
linux-arm-kernel, linux-cris-kernel, linux-fbdev, linux-fsdevel,
linux-ia64, linux-kernel, linux-media, linux-mips, linux-rdma,
linux-s390, linux-samsung-soc, linux-scsi, linux-security-module,
linux-sh
In-Reply-To: <5807A427.7010200@linux.intel.com>
On Wed 19-10-16 09:49:43, Dave Hansen wrote:
> On 10/19/2016 02:07 AM, Michal Hocko wrote:
> > On Wed 19-10-16 09:58:15, Lorenzo Stoakes wrote:
> >> On Tue, Oct 18, 2016 at 05:30:50PM +0200, Michal Hocko wrote:
> >>> I am wondering whether we can go further. E.g. it is not really clear to
> >>> me whether we need an explicit FOLL_REMOTE when we can in fact check
> >>> mm != current->mm and imply that. Maybe there are some contexts which
> >>> wouldn't work, I haven't checked.
> >>
> >> This flag is set even when /proc/self/mem is used. I've not looked deeply into
> >> this flag but perhaps accessing your own memory this way can be considered
> >> 'remote' since you're not accessing it directly. On the other hand, perhaps this
> >> is just mistaken in this case?
> >
> > My understanding of the flag is quite limited as well. All I know it is
> > related to protection keys and it is needed to bypass protection check.
> > See arch_vma_access_permitted. See also 1b2ee1266ea6 ("mm/core: Do not
> > enforce PKEY permissions on remote mm access").
>
> Yeah, we need the flag to tell us when PKEYs should be applied or not.
> The current task's PKRU (pkey rights register) should really only be
> used to impact access to the task's memory, but has no bearing on how a
> given task should access remote memory.
The question I had earlier was whether this has to be an explicit FOLL
flag used by g-u-p users or we can just use it internally when mm !=
current->mm
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH 00/10] mm: adjust get_user_pages* functions to explicitly pass FOLL_* flags
From: Dave Hansen @ 2016-10-19 16:49 UTC (permalink / raw)
To: Michal Hocko, Lorenzo Stoakes
Cc: linux-mm, Linus Torvalds, Jan Kara, Hugh Dickins, Rik van Riel,
Mel Gorman, Andrew Morton, adi-buildroot-devel, ceph-devel,
dri-devel, intel-gfx, kvm, linux-alpha, linux-arm-kernel,
linux-cris-kernel, linux-fbdev, linux-fsdevel, linux-ia64,
linux-kernel, linux-media, linux-mips, linux-rdma, linux-s390,
linux-samsung-soc, linux-scsi
In-Reply-To: <20161019090727.GE7517@dhcp22.suse.cz>
On 10/19/2016 02:07 AM, Michal Hocko wrote:
> On Wed 19-10-16 09:58:15, Lorenzo Stoakes wrote:
>> On Tue, Oct 18, 2016 at 05:30:50PM +0200, Michal Hocko wrote:
>>> I am wondering whether we can go further. E.g. it is not really clear to
>>> me whether we need an explicit FOLL_REMOTE when we can in fact check
>>> mm != current->mm and imply that. Maybe there are some contexts which
>>> wouldn't work, I haven't checked.
>>
>> This flag is set even when /proc/self/mem is used. I've not looked deeply into
>> this flag but perhaps accessing your own memory this way can be considered
>> 'remote' since you're not accessing it directly. On the other hand, perhaps this
>> is just mistaken in this case?
>
> My understanding of the flag is quite limited as well. All I know it is
> related to protection keys and it is needed to bypass protection check.
> See arch_vma_access_permitted. See also 1b2ee1266ea6 ("mm/core: Do not
> enforce PKEY permissions on remote mm access").
Yeah, we need the flag to tell us when PKEYs should be applied or not.
The current task's PKRU (pkey rights register) should really only be
used to impact access to the task's memory, but has no bearing on how a
given task should access remote memory.
^ permalink raw reply
* Re: iscsi_trx going into D state
From: Robert LeBlanc @ 2016-10-19 16:41 UTC (permalink / raw)
To: Nicholas A. Bellinger
Cc: Zhu Lingshan, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
linux-scsi-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1476858359.8490.97.camel-XoQW25Eq2zviZyQQd+hFbcojREIfoBdhmpATvIKMPHk@public.gmane.org>
Nicholas,
I didn't have high hopes for the patch because we were not seeing
TMR_ABORT_TASK (or 'abort') in dmesg or /var/log/messages, but it
seemed to help regardless. Our clients finally OOMed from the hung
sessions, so we are having to reboot them and we will do some more
testing. We haven't put the updated kernel on our clients yet. Our
clients have iSCSI root disks so I'm not sure if we can get a vmcore
on those, but we will do what we can to get you a vmcore from the
target if it happens again.
As far as our configuration: It is a superMicro box with 6 SAMSUNG
MZ7LM3T8HCJM-00005 SSDs. Two are for root and four are in mdadm
RAID-10 for exporting via iSCSI/iSER. We have ZFS on top of the
RAID-10 for checksum and snapshots only and we export ZVols to the
clients (one or more per VM on the client). We do not persist the
export info (targetcli saveconfig), but regenerate it from scripts.
The client receives two or more of these exports and puts them in a
RAID-1 device. The exports are served by iSER one one port and also by
normal iSCSI on a different port for compatibility, but not normally
used. If you need more info about the config, please let me know. It
was kind of a vague request so I'm not sure what exactly is important
to you.
Thanks for helping us with this,
Robert LeBlanc
When we have problems, we usually see this in the logs:
Oct 17 08:57:50 prv-0-12-sanstack kernel: iSCSI Login timeout on
Network Portal 0.0.0.0:3260
Oct 17 08:57:50 prv-0-12-sanstack kernel: Unexpected ret: -104 send data 48
Oct 17 08:57:50 prv-0-12-sanstack kernel: tx_data returned -32, expecting 48.
Oct 17 08:57:50 prv-0-12-sanstack kernel: iSCSI Login negotiation failed.
I found some backtraces in the logs, not sure if this is helpful, this
is before your patch (your patch booted at Oct 18 10:36:59):
Oct 17 15:43:12 prv-0-12-sanstack kernel: INFO: rcu_sched
self-detected stall on CPU
Oct 17 15:43:12 prv-0-12-sanstack kernel: #0115-...: (41725 ticks this
GP) idle=b59/140000000000001/0 softirq=535/535 fqs=30992
Oct 17 15:43:12 prv-0-12-sanstack kernel: #011 (t=42006 jiffies g=1550
c=1549 q=0)
Oct 17 15:43:12 prv-0-12-sanstack kernel: Task dump for CPU 5:
Oct 17 15:43:12 prv-0-12-sanstack kernel: kworker/u68:2 R running
task 0 17967 2 0x00000008
Oct 17 15:43:12 prv-0-12-sanstack kernel: Workqueue: isert_comp_wq
isert_cq_work [ib_isert]
Oct 17 15:43:12 prv-0-12-sanstack kernel: ffff883f4c0dca80
00000000af8ca7a4 ffff883f7fb43da8 ffffffff810ac83f
Oct 17 15:43:12 prv-0-12-sanstack kernel: 0000000000000005
ffffffff81adb680 ffff883f7fb43dc0 ffffffff810af179
Oct 17 15:43:12 prv-0-12-sanstack kernel: 0000000000000006
ffff883f7fb43df0 ffffffff810e1c10 ffff883f7fb57b80
Oct 17 15:43:12 prv-0-12-sanstack kernel: Call Trace:
Oct 17 15:43:12 prv-0-12-sanstack kernel: <IRQ> [<ffffffff810ac83f>]
sched_show_task+0xaf/0x110
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff810af179>]
dump_cpu_task+0x39/0x40
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff810e1c10>]
rcu_dump_cpu_stacks+0x80/0xb0
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff810e6040>]
rcu_check_callbacks+0x540/0x820
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff810afd51>] ?
account_system_time+0x81/0x110
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff810fa9a0>] ?
tick_sched_do_timer+0x50/0x50
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff810eb4d9>]
update_process_times+0x39/0x60
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff810fa755>]
tick_sched_handle.isra.17+0x25/0x60
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff810fa9dd>]
tick_sched_timer+0x3d/0x70
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff810ec0c2>]
__hrtimer_run_queues+0x102/0x290
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff810ec5a8>]
hrtimer_interrupt+0xa8/0x1a0
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff81052c65>]
local_apic_timer_interrupt+0x35/0x60
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff8172343d>]
smp_apic_timer_interrupt+0x3d/0x50
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff817216f7>]
apic_timer_interrupt+0x87/0x90
Oct 17 15:43:12 prv-0-12-sanstack kernel: <EOI> [<ffffffff810d70fe>]
? console_unlock+0x41e/0x4e0
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff810d74bc>]
vprintk_emit+0x2fc/0x500
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff810d783f>]
vprintk_default+0x1f/0x30
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff81174c2a>] printk+0x5d/0x74
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff814bc351>]
transport_lookup_cmd_lun+0x1d1/0x200
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff814edcf0>]
iscsit_setup_scsi_cmd+0x230/0x540
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffffa0890bf3>]
isert_rx_do_work+0x3f3/0x7f0 [ib_isert]
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffffa0891174>]
isert_cq_work+0x184/0x770 [ib_isert]
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff8109734f>]
process_one_work+0x14f/0x400
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff81097bc4>]
worker_thread+0x114/0x470
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff8171c55a>] ?
__schedule+0x34a/0x7f0
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff81097ab0>] ?
rescuer_thread+0x310/0x310
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff8109d708>] kthread+0xd8/0xf0
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff8109d630>] ?
kthread_park+0x60/0x60
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff81720c8f>]
ret_from_fork+0x3f/0x70
Oct 17 15:43:12 prv-0-12-sanstack kernel: [<ffffffff8109d630>] ?
kthread_park+0x60/0x60
Oct 17 16:34:03 prv-0-12-sanstack kernel: INFO: rcu_sched
self-detected stall on CPU
Oct 17 16:34:03 prv-0-12-sanstack kernel: #01128-...: (5999 ticks this
GP) idle=2f9/140000000000001/0 softirq=457/457 fqs=4830
Oct 17 16:34:03 prv-0-12-sanstack kernel: #011 (t=6000 jiffies g=3546
c=3545 q=0)
Oct 17 16:34:03 prv-0-12-sanstack kernel: Task dump for CPU 28:
Oct 17 16:34:03 prv-0-12-sanstack kernel: iscsi_np R running
task 0 16597 2 0x0000000c
Oct 17 16:34:03 prv-0-12-sanstack kernel: ffff887f40350000
00000000b98a67bb ffff887f7f503da8 ffffffff810ac8ff
Oct 17 16:34:03 prv-0-12-sanstack kernel: 000000000000001c
ffffffff81adb680 ffff887f7f503dc0 ffffffff810af239
Oct 17 16:34:03 prv-0-12-sanstack kernel: 000000000000001d
ffff887f7f503df0 ffffffff810e1cd0 ffff887f7f517b80
Oct 17 16:34:03 prv-0-12-sanstack kernel: Call Trace:
Oct 17 16:34:03 prv-0-12-sanstack kernel: <IRQ> [<ffffffff810ac8ff>]
sched_show_task+0xaf/0x110
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff810af239>]
dump_cpu_task+0x39/0x40
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff810e1cd0>]
rcu_dump_cpu_stacks+0x80/0xb0
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff810e6100>]
rcu_check_callbacks+0x540/0x820
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff810afe11>] ?
account_system_time+0x81/0x110
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff810faa60>] ?
tick_sched_do_timer+0x50/0x50
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff810eb599>]
update_process_times+0x39/0x60
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff810fa815>]
tick_sched_handle.isra.17+0x25/0x60
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff810faa9d>]
tick_sched_timer+0x3d/0x70
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff810ec182>]
__hrtimer_run_queues+0x102/0x290
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff810ec668>]
hrtimer_interrupt+0xa8/0x1a0
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff81052c65>]
local_apic_timer_interrupt+0x35/0x60
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff81723cbd>]
smp_apic_timer_interrupt+0x3d/0x50
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff81721f77>]
apic_timer_interrupt+0x87/0x90
Oct 17 16:34:03 prv-0-12-sanstack kernel: <EOI> [<ffffffff810d71be>]
? console_unlock+0x41e/0x4e0
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff810d757c>]
vprintk_emit+0x2fc/0x500
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff810d78ff>]
vprintk_default+0x1f/0x30
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff81174dde>] printk+0x5d/0x74
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff814e71ad>]
iscsi_target_locate_portal+0x62d/0x6f0
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff814e5100>]
iscsi_target_login_thread+0x6f0/0xfc0
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff814e4a10>] ?
iscsi_target_login_sess_out+0x250/0x250
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff8109d7c8>] kthread+0xd8/0xf0
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff8109d6f0>] ?
kthread_park+0x60/0x60
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff8172150f>]
ret_from_fork+0x3f/0x70
Oct 17 16:34:03 prv-0-12-sanstack kernel: [<ffffffff8109d6f0>] ?
kthread_park+0x60/0x60
I don't think this one is related, but it happened a couple of times:
Oct 17 11:46:52 prv-0-12-sanstack kernel: INFO: rcu_sched
self-detected stall on CPU
Oct 17 11:46:52 prv-0-12-sanstack kernel: #01119-...: (5999 ticks this
GP) idle=727/140000000000001/0 softirq=1346/1346 fqs=4990
Oct 17 11:46:52 prv-0-12-sanstack kernel: #011 (t=6000 jiffies g=4295
c=4294 q=0)
Oct 17 11:46:52 prv-0-12-sanstack kernel: Task dump for CPU 19:
Oct 17 11:46:52 prv-0-12-sanstack kernel: kworker/19:1 R running
task 0 301 2 0x00000008
Oct 17 11:46:52 prv-0-12-sanstack kernel: Workqueue:
events_power_efficient fb_flashcursor
Oct 17 11:46:52 prv-0-12-sanstack kernel: ffff883f6009ca80
00000000010a7cdd ffff883f7fcc3da8 ffffffff810ac8ff
Oct 17 11:46:52 prv-0-12-sanstack kernel: 0000000000000013
ffffffff81adb680 ffff883f7fcc3dc0 ffffffff810af239
Oct 17 11:46:52 prv-0-12-sanstack kernel: 0000000000000014
ffff883f7fcc3df0 ffffffff810e1cd0 ffff883f7fcd7b80
Oct 17 11:46:52 prv-0-12-sanstack kernel: Call Trace:
Oct 17 11:46:52 prv-0-12-sanstack kernel: <IRQ> [<ffffffff810ac8ff>]
sched_show_task+0xaf/0x110
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff810af239>]
dump_cpu_task+0x39/0x40
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff810e1cd0>]
rcu_dump_cpu_stacks+0x80/0xb0
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff810e6100>]
rcu_check_callbacks+0x540/0x820
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff810afe11>] ?
account_system_time+0x81/0x110
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff810faa60>] ?
tick_sched_do_timer+0x50/0x50
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff810eb599>]
update_process_times+0x39/0x60
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff810fa815>]
tick_sched_handle.isra.17+0x25/0x60
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff810faa9d>]
tick_sched_timer+0x3d/0x70
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff810ec182>]
__hrtimer_run_queues+0x102/0x290
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff810ec668>]
hrtimer_interrupt+0xa8/0x1a0
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff81052c65>]
local_apic_timer_interrupt+0x35/0x60
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff81723cbd>]
smp_apic_timer_interrupt+0x3d/0x50
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff81721f77>]
apic_timer_interrupt+0x87/0x90
Oct 17 11:46:52 prv-0-12-sanstack kernel: <EOI> [<ffffffff810d71be>]
? console_unlock+0x41e/0x4e0
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff813866ad>]
fb_flashcursor+0x5d/0x140
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff8138bc00>] ?
bit_clear+0x110/0x110
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff8109740f>]
process_one_work+0x14f/0x400
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff81097c84>]
worker_thread+0x114/0x470
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff8171cdda>] ?
__schedule+0x34a/0x7f0
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff81097b70>] ?
rescuer_thread+0x310/0x310
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff8109d7c8>] kthread+0xd8/0xf0
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff8109d6f0>] ?
kthread_park+0x60/0x60
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff8172150f>]
ret_from_fork+0x3f/0x70
Oct 17 11:46:52 prv-0-12-sanstack kernel: [<ffffffff8109d6f0>] ?
kthread_park+0x60/0x60
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Wed, Oct 19, 2016 at 12:25 AM, Nicholas A. Bellinger
<nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org> wrote:
> On Tue, 2016-10-18 at 16:13 -0600, Robert LeBlanc wrote:
>> Nicholas,
>>
>> We patched this in and for the first time in many reboots, we didn't
>> have iSCSI going straight into D state. We have had to work on a
>> couple of other things, so we don't know if this is just a coincidence
>> or not. We will reboot back into the old kernel and back a few times
>> and do some more testing, but so far it has given us a little bit of
>> hope that we may be narrowing down on the root cause. We will report
>> back once we have some more info.
>>
>> Thank you,
>> Robert LeBlanc
>> ----------------
>> Robert LeBlanc
>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
>>
>
> Hello Robert,
>
> Thanks for the update. Btw, if the original /var/log/messages
> reproduction logs for iser-target are still handy, I'm happy to have
> a look to confirm. Feel free to send them along here, or off-list if
> necessary.
>
> For further reference, you can also enable Linux kernel crash dump
> (LKCD) at build time using CONFIG_CRASH_DUMP=y, so it's possible to
> manually generate a vmcore dumpfile of the running system via 'echo c
>> /proc/sysrq-trigger', once the bug occurs.
>
> http://cateee.net/lkddb/web-lkddb/CRASH_DUMP.html
>
> Note in order to fully debug within this in a LKCD environment, it
> requires the vmcore dump from /var/crash/, unstripped vmlinux,
> target_core_mod, iscsi_target_mod and ib_isert modules matching the
> specific particular x86_64 build setup of the running system.
>
> Also, can you share a bit more about the details of your particular
> iser-target + backend setup..?
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox