[PATCH v4] blktrace: Fix potentail deadlock between delete & sysfs ops

[PATCH v4] blktrace: Fix potentail deadlock between delete & sysfs ops

[Waiman Long]

The lockdep code had reported the following unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(s_active#228);
                               lock(&bdev->bd_mutex/1);
                               lock(s_active#228);
  lock(&bdev->bd_mutex);

 *** DEADLOCK ***

The deadlock may happen when one task (CPU1) is trying to delete a
partition in a block device and another task (CPU0) is accessing
tracing sysfs file (e.g. /sys/block/dm-1/trace/act_mask) in that
partition.

The s_active isn't an actual lock. It is a reference count (kn->count)
on the sysfs (kernfs) file. Removal of a sysfs file, however, require
a wait until all the references are gone. The reference count is
treated like a rwsem using lockdep instrumentation code.

The fact that a thread is in the sysfs callback method or in the
ioctl call means there is a reference to the opended sysfs or device
file. That should prevent the underlying block structure from being
removed.

Instead of using bd_mutex, a new global blktrace mutex is now used
to protect against concurrent access, creation and destruction of the
blk_trace structure that is used only in the blktrace.c file. As
blktrace files will not be frequently accessed, using a global mutex
should not cause any performance problem.

Signed-off-by: Waiman Long <longman@redhat.com>
---

 v4:
  - Use blktrace_mutex in blk_trace_ioctl() as well.

 v3:
  - Use a global blktrace_mutex to serialize sysfs attribute accesses
    instead of the bd_mutex. 

 v2:
  - Use READ_ONCE() and smp_store_mb() to read and write bd_deleting.
  - Check for signal in the mutex_trylock loops.
  - Use usleep() instead of schedule() for RT tasks.

 kernel/trace/blktrace.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)

diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
index bc364f8..ec5a919 100644
--- a/kernel/trace/blktrace.c
+++ b/kernel/trace/blktrace.c
@@ -624,6 +624,17 @@ int blk_trace_startstop(struct request_queue *q, int start)
 }
 EXPORT_SYMBOL_GPL(blk_trace_startstop);
 
+/*
+ * The bd_mutex was used previously for protecting blk_trace structure.
+ * That could lead to deadlock with concurrent block device deletion and
+ * sysfs access. So a global blktrace_mutex is now used instead for
+ * protecting the blk_trace structure.
+ *
+ * The references to the opened sysfs or device files should prevent the
+ * underlying block device from being removed.
+ */
+static DEFINE_MUTEX(blktrace_mutex);
+
 /**
  * blk_trace_ioctl: - handle the ioctls associated with tracing
  * @bdev:	the block device
@@ -641,7 +652,7 @@ int blk_trace_ioctl(struct block_device *bdev, unsigned cmd, char __user *arg)
 	if (!q)
 		return -ENXIO;
 
-	mutex_lock(&bdev->bd_mutex);
+	mutex_lock(&blktrace_mutex);
 
 	switch (cmd) {
 	case BLKTRACESETUP:
@@ -667,7 +678,7 @@ int blk_trace_ioctl(struct block_device *bdev, unsigned cmd, char __user *arg)
 		break;
 	}
 
-	mutex_unlock(&bdev->bd_mutex);
+	mutex_unlock(&blktrace_mutex);
 	return ret;
 }
 
@@ -1622,7 +1633,7 @@ static ssize_t sysfs_blk_trace_attr_show(struct device *dev,
 	if (q == NULL)
 		goto out_bdput;
 
-	mutex_lock(&bdev->bd_mutex);
+	mutex_lock(&blktrace_mutex);
 
 	if (attr == &dev_attr_enable) {
 		ret = sprintf(buf, "%u\n", !!q->blk_trace);
@@ -1641,7 +1652,7 @@ static ssize_t sysfs_blk_trace_attr_show(struct device *dev,
 		ret = sprintf(buf, "%llu\n", q->blk_trace->end_lba);
 
 out_unlock_bdev:
-	mutex_unlock(&bdev->bd_mutex);
+	mutex_unlock(&blktrace_mutex);
 out_bdput:
 	bdput(bdev);
 out:
@@ -1683,7 +1694,7 @@ static ssize_t sysfs_blk_trace_attr_store(struct device *dev,
 	if (q == NULL)
 		goto out_bdput;
 
-	mutex_lock(&bdev->bd_mutex);
+	mutex_lock(&blktrace_mutex);
 
 	if (attr == &dev_attr_enable) {
 		if (value)
@@ -1709,7 +1720,7 @@ static ssize_t sysfs_blk_trace_attr_store(struct device *dev,
 	}
 
 out_unlock_bdev:
-	mutex_unlock(&bdev->bd_mutex);
+	mutex_unlock(&blktrace_mutex);
 out_bdput:
 	bdput(bdev);
 out:
-- 
1.8.3.1

Re: [PATCH v4] blktrace: Fix potentail deadlock between delete & sysfs ops

[Bart Van Assche]

T24gRnJpLCAyMDE3LTA4LTE4IGF0IDE2OjAxIC0wNDAwLCBXYWltYW4gTG9uZyB3cm90ZToNCj4g
WyAuLi4gXQ0KPiBJbnN0ZWFkIG9mIHVzaW5nIGJkX211dGV4LCBhIG5ldyBnbG9iYWwgYmxrdHJh
Y2UgbXV0ZXggaXMgbm93IHVzZWQNCj4gdG8gcHJvdGVjdCBhZ2FpbnN0IGNvbmN1cnJlbnQgYWNj
ZXNzLCBjcmVhdGlvbiBhbmQgZGVzdHJ1Y3Rpb24gb2YgdGhlDQo+IGJsa190cmFjZSBzdHJ1Y3R1
cmUgdGhhdCBpcyB1c2VkIG9ubHkgaW4gdGhlIGJsa3RyYWNlLmMgZmlsZS4gQXMNCj4gYmxrdHJh
Y2UgZmlsZXMgd2lsbCBub3QgYmUgZnJlcXVlbnRseSBhY2Nlc3NlZCwgdXNpbmcgYSBnbG9iYWwg
bXV0ZXgNCj4gc2hvdWxkIG5vdCBjYXVzZSBhbnkgcGVyZm9ybWFuY2UgcHJvYmxlbS4NCj4gWyAu
Li4gXQ0KPiArLyoNCj4gKyAqIFRoZSBiZF9tdXRleCB3YXMgdXNlZCBwcmV2aW91c2x5IGZvciBw
cm90ZWN0aW5nIGJsa190cmFjZSBzdHJ1Y3R1cmUuDQo+ICsgKiBUaGF0IGNvdWxkIGxlYWQgdG8g
ZGVhZGxvY2sgd2l0aCBjb25jdXJyZW50IGJsb2NrIGRldmljZSBkZWxldGlvbiBhbmQNCj4gKyAq
IHN5c2ZzIGFjY2Vzcy4gU28gYSBnbG9iYWwgYmxrdHJhY2VfbXV0ZXggaXMgbm93IHVzZWQgaW5z
dGVhZCBmb3INCj4gKyAqIHByb3RlY3RpbmcgdGhlIGJsa190cmFjZSBzdHJ1Y3R1cmUuDQo+ICsg
Kg0KPiArICogVGhlIHJlZmVyZW5jZXMgdG8gdGhlIG9wZW5lZCBzeXNmcyBvciBkZXZpY2UgZmls
ZXMgc2hvdWxkIHByZXZlbnQgdGhlDQo+ICsgKiB1bmRlcmx5aW5nIGJsb2NrIGRldmljZSBmcm9t
IGJlaW5nIHJlbW92ZWQuDQo+ICsgKi8NCj4gK3N0YXRpYyBERUZJTkVfTVVURVgoYmxrdHJhY2Vf
bXV0ZXgpOw0KDQpIZWxsbyBXYWltYW4sDQoNClRoYW5rcyBmb3IgaGF2aW5nIGFkZHJlc3NlZCBt
eSBwcmV2aW91cyBjb21tZW50LiBSZWdhcmRpbmcgdGhpcyBwYXRjaDogc29ycnkNCmJ1dCBJIGRv
bid0IHRoaW5rIGl0J3MgYSBnb29kIGlkZWEgdG8gdXNlIGEgZ2xvYmFsIG11dGV4IGZvciBzZXJp
YWxpemluZw0KYWNjZXNzZXMgdG8gdHJhY2luZyBkYXRhIG9mIGEgc2luZ2xlIGJsb2NrIGRldmlj
ZS4gR2xvYmFsIG11dGV4ZXMgY3JlYXRlDQp1bndhbnRlZCBsb2NrIGRlcGVuZGVuY2llcyBiZXR3
ZWVuIGRpZmZlcmVudCBibG9jayBkZXZpY2VzLiBBZGRpdGlvbmFsbHksIG9uDQptdWx0aXByb2Nl
c3NvciBzeXN0ZW1zIGdsb2JhbCBtdXRleGVzIGNhbiBjYXVzZSBjYWNoZSBsaW5lIHBpbmctcG9u
ZyBiZXR3ZWVuDQpwcm9jZXNzb3JzIGFuZCBoZW5jZSBjYW4gY2F1c2UgYSBzZXZlcmUgc2xvd2Rv
d24uIFBsZWFzZSBtYWtlIGJsa3RyYWNlX211dGV4DQpwZXIgYmxvY2sgZGV2aWNlIGluc3RlYWQg
b2YgZ2xvYmFsLg0KDQpUaGFua3MsDQoNCkJhcnQu

Re: [PATCH v4] blktrace: Fix potentail deadlock between delete & sysfs ops

[Waiman Long]

On 08/18/2017 04:18 PM, Bart Van Assche wrote:
> On Fri, 2017-08-18 at 16:01 -0400, Waiman Long wrote:
>> [ ... ]
>> Instead of using bd_mutex, a new global blktrace mutex is now used
>> to protect against concurrent access, creation and destruction of the
>> blk_trace structure that is used only in the blktrace.c file. As
>> blktrace files will not be frequently accessed, using a global mutex
>> should not cause any performance problem.
>> [ ... ]
>> +/*
>> + * The bd_mutex was used previously for protecting blk_trace structur=
e.
>> + * That could lead to deadlock with concurrent block device deletion =
and
>> + * sysfs access. So a global blktrace_mutex is now used instead for
>> + * protecting the blk_trace structure.
>> + *
>> + * The references to the opened sysfs or device files should prevent =
the
>> + * underlying block device from being removed.
>> + */
>> +static DEFINE_MUTEX(blktrace_mutex);
> Hello Waiman,
>
> Thanks for having addressed my previous comment. Regarding this patch: =
sorry
> but I don't think it's a good idea to use a global mutex for serializin=
g
> accesses to tracing data of a single block device. Global mutexes creat=
e
> unwanted lock dependencies between different block devices. Additionall=
y, on
> multiprocessor systems global mutexes can cause cache line ping-pong be=
tween
> processors and hence can cause a severe slowdown. Please make blktrace_=
mutex
> per block device instead of global.

I fully understand the problem of a global lock. The main reason of
using a global lock here is that the blktrace APIs are not in a
performance critical path. In fact, I think it is used primarily for
debugging purpose. Activating it will certainly slow thing down no
matter what kind of lock is used.

I also don't believe that the blktrace APIs will be used in a high
enough frequency that it will cause a performance issue. Please let me
know if I am wrong in my assumptions.

Cheers,
Longman