mount command stuck at synchronize_rcu on a RT patched kernel.

mount command stuck at synchronize_rcu on a RT patched kernel.

[Mahesh Sivapuram]

Hello All,

We are facing an issue with 3.10.65 kernel + RT69 patch on a 12core xeon.
We are running real time applications (higher priority than kernel threads,
requiring the resolution of 1ms or less) on ioslated CPU's which are
configured through kernel boot arguments as indicated below. Also, we have
offloaded the RCU call backs to other book keeping CPU cores. But when we
try to perform some general commands like mount some directory over NFS,
the command hangs and never completes till we stop our real time
applications. The moment real time applications are stopped, command
finishes immediately.

When we see the backtrace of the mount command, it waiting for
"synchronize_rcu". Since we have already offloaded RCU call backs, we
expect nothing to be done on isolcpus except rcuc process on isolcpus
waking the offloaded threads on other cores.

Can someone help us understand what kernel parameters/config needs to be
modified/tuned to achieve the proper behavior of the system ?

Thanks in advance.

Backtrace:
----------
[root at localhost ~]# ps -eaf | grep mount
root     17915  4532  0 19:13 pts/0    00:00:00 umount /bng4
root     17916 17915  0 19:13 pts/0    00:00:00 /sbin/umount.nfs /bng4
root     17962 17945  0 19:14 pts/2    00:00:00 grep --color=auto mount
[root at localhost ~]# cat /proc/17916/stack
[<ffffffff810612b4>] wait_rcu_gp+0x44/0x50
[<ffffffff810c6ec9>] synchronize_rcu+0x29/0x40
[<ffffffff8126446a>] nfs_free_server+0xea/0x180
[<ffffffff8126e49e>] nfs_kill_super+0x1e/0x30
[<ffffffff8115a4ed>] deactivate_locked_super+0x3d/0x60
[<ffffffff8115ab16>] deactivate_super+0x46/0x60
[<ffffffff811770ef>] mntput_no_expire+0xef/0x150
[<ffffffff81178376>] SyS_umount+0x96/0x3d0
[<ffffffff818cbb46>] system_call_fastpath+0x1a/0x1f
[<ffffffffffffffff>] 0xffffffffffffffff
[root at localhost ~]#

Boot arguments:
----------------
[root at localhost ~]# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.10.65-rt69 raid=noautodetect console=tty0,115200
ip=none root=/dev/ram0 ramdisk_size=12582912 rw clock=pit rcu_nocbs=1-8
rcu_nocb_poll isolcpus=1-8 irqaffinity=0 tsc=perfect selinux=0 enforcing=0
intel_iommu=on iommu=pt default_hugepagesz=2M hugepagesz=2M hugepages=4096
nohz_full=1-8 idle=poll noswap pci=pcie_bus_perf pcie_aspm=off
[root at localhost ~]#

Below is our RCU config while compiling the kernel + rt patch.

# RCU Subsystem
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_RCU_STALL_COMMON=y
# CONFIG_RCU_USER_QS is not set
CONFIG_RCU_FANOUT=64
CONFIG_RCU_FANOUT_LEAF=16
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_RCU_BOOST=y
CONFIG_RCU_BOOST_PRIO=85
CONFIG_RCU_BOOST_DELAY=500
CONFIG_RCU_NOCB_CPU=y
CONFIG_RCU_NOCB_CPU_NONE=y
# CONFIG_RCU_NOCB_CPU_ZERO is not set
# CONFIG_RCU_NOCB_CPU_ALL is not set
# RCU Debugging
# CONFIG_PROVE_RCU_DELAY is not set
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=21
CONFIG_RCU_CPU_STALL_VERBOSE=y
CONFIG_RCU_CPU_STALL_INFO=y
# CONFIG_RCU_TRACE is not set

Thanks & Regards
Mahesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20180604/490ec418/attachment-0001.html>

mount command stuck at synchronize_rcu on a RT patched kernel.

[Greg KH]

On Mon, Jun 04, 2018 at 03:55:34PM +0530, Mahesh Sivapuram wrote:
> Hello All,
> 
> We are facing an issue with 3.10.65 kernel + RT69 patch on a 12core xeon.

First off, that is a very old and "obsolete" kernel version.  As you are
probably stuck at that version due to a support contract, I strongly
suggest you get support from the company you are paying that money to,
as they are the only ones that really know what is in that kernel.
There's almost nothing the community can do to help you out, sorry.

> We are running real time applications (higher priority than kernel threads,
> requiring the resolution of 1ms or less) on ioslated CPU's which are
> configured through kernel boot arguments as indicated below. Also, we have
> offloaded the RCU call backs to other book keeping CPU cores. But when we
> try to perform some general commands like mount some directory over NFS,
> the command hangs and never completes till we stop our real time
> applications. The moment real time applications are stopped, command
> finishes immediately.
> 
> When we see the backtrace of the mount command, it waiting for
> "synchronize_rcu". Since we have already offloaded RCU call backs, we
> expect nothing to be done on isolcpus except rcuc process on isolcpus
> waking the offloaded threads on other cores.

Messing with where RCU callbacks run from, is a _very_ risky thing to
do.  There are built-in assumptions that RCU uses to be able to do its
work properly, and some of those are that it can touch specific CPUs.

Also, I don't think the 3.10.y kernel had all of the "isolate cpus"
logic properly implemented in it, I would strongly suggest moving to a
newer kernel version (4.17?) that way if things still do not work, the
community has a chance to provide some help.

good luck!

greg k-h