From mboxrd@z Thu Jan  1 00:00:00 1970
From: Hitoshi Mitake <mitake.hitoshi@gmail.com>
Subject: Re: [PATCH v3] tgtd: refresh ready fds of event loop after event deletion
Date: Thu, 06 Mar 2014 15:54:03 +0900
Message-ID: <87fvmv4y2c.wl%mitake.hitoshi@lab.ntt.co.jp>
References: <1393391602-6801-1-git-send-email-mitake.hitoshi@lab.ntt.co.jp>
	<5315EDE1.80103@mellanox.com>
Mime-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka")
Return-path: <stgt-owner@vger.kernel.org>
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=date:message-id:from:to:cc:subject:in-reply-to:references
         :user-agent:mime-version:content-type;
        bh=B1GAt8HMF3l2iR77hYfypMa5PUjcXBRsBpYmsZaOUwg=;
        b=WSgnrtqhNRQ6+HyTFzNVjdDxHaRLFc+BgckAZcBZf8MTNCIpvZkQ8q3Wf/jOIAQ7NH
         fHhIkGeK4S1UfQ1MplMKXCOuWnSQF4jWe3Y2TLrOPHpOXChzV0mtUv76cS40+IK4uW6H
         6u+svT55vMJ3H0iZrkzD0WVYi8V4G7x7oMwSwgOJvmA4lInfvRhoKeYEK6o080k67Axt
         m90qfe0VxqXVtbkAr5AIHkIVtkEzdSI2XmGrRyOnlrS0fLU3xvqDqsraRpE8+UJJHFOv
         gQF8w9PluSx7GDFrYR7/4Kl8Z4m1P+Zr+Dxdoqi12Fbl1g+FQusEILaPkZfqbMZ/m/u9
         o1nA==
In-Reply-To: <5315EDE1.80103@mellanox.com>
Sender: stgt-owner@vger.kernel.org
List-ID: <stgt.vger.kernel.org>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
To: Or Gerlitz <ogerlitz@mellanox.com>
Cc: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>, stgt@vger.kernel.org, mitake.hitoshi@gmail.com, Roi Dayan <roid@mellanox.com>

At Tue, 4 Mar 2014 17:14:41 +0200,
Or Gerlitz wrote:
> 
> On 26/02/2014 07:13, Hitoshi Mitake wrote:
> > For example, we can produce segfault of tgtd under heavy load. Below
> > is a backtrace obtained from the core file:
> >   (gdb) bt
> >   #0  0x0000000000000000 in ?? ()
> >   #1  0x0000000000411419 in event_loop () at tgtd.c:414
> >   #2  0x0000000000411b65 in main (argc=<value optimized out>, argv=<value optimized out>) at tgtd.c:591
> >
> > To be honest, I still don't find an event handler which calls
> > tgt_event_del() for other fds. But with this modification, the above
> > segfault is avoided. The change seems to be effective.
> 
> Just want to make sure I follow --- you do have a way to reproduce the 
> bug, but from code inspection you didn't find an event handler in tgt 
> which calls  tgt_event_del() for "other" fds which is the trigger for 
> the bug, right?

Yes. To be more precise, I'd like to describe my understanding:

1. There are some event handlers which can close "other
fds". e.g. mtask_recv_send_handler(). It can be invoked via input of
unix domain socket and close fds of tcp connections when user invokes
"--op delete --mode target".

2. But we can produce the above segfault without using it...

> 
> Can you please provide the steps to reproduce the bug?

We are using 4 node cluster connected via 10Gbps ethernet. 1 node
executes tgtd for providing iSCSI target. The backing store is
sheepdog. All nodes execute 4 VMs and read iso file (about 4GB) from
single logical unit.

# The test is mocking an environment of thin clients. We need multiple
# dd processes for exhausting the 10Gbps network.

When we run the above test several times, the segfault occurs. But, of
course, we don't invoke the "tgtadm --op delete --mode target" during
the testing.

Thanks,
Hitoshi