From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 54ADC3C1994 for ; Thu, 7 May 2026 12:27:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778156880; cv=none; b=mWyrylitM2msjnR65AUcR3JYeMNlJqDxRnnv7/FFmSXPZQ+yzk6h5tTawKgSIOSAEgT0X0YEp2IaI9+HbF+OBV4s8qMbyFJ6+cU5wDwVrRTWjzbHC13RaIPYPAisiGYh95pP884mrbXokcH0Lx9lBU6d17iNwWA8Wt17+H5juBc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778156880; c=relaxed/simple; bh=mry7fPSwDpLlrYEx/LJ8bD4menn0NAgWZBGmTokNjUw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iM91IGbP41W3p2IAEvSYjW1Cqma3ceUO/j6M46LZBQy1X5YXp9WlaUMI8wviyHSfZYpWJjWlHEwg1e7X74kf9QEDe1m1151bCiqS160jzAkxWtYKEm/fWAdgM72nVVKT316t+MFDnYbyKiMSH35nChBc0I0gutXbdrfcMz/BLWo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Y5EGiFik; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=KR8f4Wbk; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Y5EGiFik"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="KR8f4Wbk" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1778156877; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gdw4QjJQwh9CCkmForNjrUpru0aKscBcQPvRnVkBnYU=; b=Y5EGiFikfu4WeYdgxiMSZe4ZFc+N2fhuceYjczpZpUmERnDZKWyuA6ksFU4SpbYGNWREPS gIkE+AoRhBpvYy0aEMn3lpeXA8HqVoG3JnyPiK/lx8/At4kUK+cU3H2ijuq8NX9W7zOKos TY/+aZ+EUU6fqTMRIMNpz9QKCkxQeGo= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-504-ofSFKY05Ml2f-UH3VajCrw-1; Thu, 07 May 2026 08:27:56 -0400 X-MC-Unique: ofSFKY05Ml2f-UH3VajCrw-1 X-Mimecast-MFC-AGG-ID: ofSFKY05Ml2f-UH3VajCrw_1778156874 Received: by mail-ed1-f69.google.com with SMTP id 4fb4d7f45d1cf-67c2b480857so1316474a12.3 for ; Thu, 07 May 2026 05:27:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1778156874; x=1778761674; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gdw4QjJQwh9CCkmForNjrUpru0aKscBcQPvRnVkBnYU=; b=KR8f4WbkolA/9qs/iHwPwLbz6XOM2goOxxzriJTzJxdw84mFbkTLzej0xO55syNpAu ioK2bDR1Y306AWDlbfUy7k9IRoNrXo4DqF8+zg0FNBWxaYiUrPjsTcelDTKW4VrTg1P/ j+uun6lLXqfmUvHvlzQ3pbhunZr4eBNMMDIsM8EMYakpsbYwcN/0GLI2zmQw6Xd5b3G0 6FufYaqz5X22Y6YBfHRHogv/sY7SSmiK2rqsDbQQMPxG12Gz77ETnrS/dibaWqnbDPkl 6ZRK6lTb19fEeGH2Z7LzbXP8IKw6lrGiEMgGkvZxeBUG6S1XiScShV72jwCvSEU3MPa1 AQKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778156874; x=1778761674; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=gdw4QjJQwh9CCkmForNjrUpru0aKscBcQPvRnVkBnYU=; b=H7aatr26Mc9gPvm92l6og+Vyxebv6d1AZV2xrPW8RR94CsZkO0rP6PFUY7eL8PutNP v9rHRj5EnoLxQgo0odLrZ7g1visGEEECOmwgDBRfPFFGwVIwVRUGDknEo3gd+LCsK/f0 fIspbS7dIXUMJ+sv6uS5WFzpudkAO6+vSKWdjiWqZU9z1gG6s1law8IIVEyOpqK57Fsr hj59QR5NJFKZe7wF0QQ3xs81px6IcRN/0wYcIErSVGLbokoUC66OhRu3QnVP/6U3xW4H VMMKNlH3Q2LT1fd2UzbYYzCGxr6UJPMocJORZINaW8EMmu4vbSgQ9eP+CgcRE1oh+Q8e wi4w== X-Gm-Message-State: AOJu0YyDRvl1EL/PaRcYmR+qDBtoz9eJ9Sh4K3pq+4Kqh6wpEfMh1pLq tCHk4B6gu8DqFgTgABWuyrfW36x/NNssk0nKMqKG1hGU2YFL5imjUU2FczmrIodpHh9dN1Nhc+q 09MKctQGiq24Dc94GHBtILjfssXWlHnlWC8YNLVj30b6tsQ/6x1IiD/N8u5Ra2yLQ4o7R1Z/n6J ur X-Gm-Gg: AeBDietW+ld2huHseF7cIpH5wZ23KjZyvLw45LGiaPJAsXTM0nFukfwiMUUnb5I28bD 3B5oZo5bDqD0SiASpiU3kySYKNkEEpouIeeG3Dy9s9tDxBPWQgVZkKUbSiaQLzJ56Tx38RLZzGD gBHhBYfcRwAX7jdgoExaHsFHgxuv7MsBXKKakPYCFJszTUHRyQZaPQehpeB97+CCRqHtWPIks3W ckmEgFNVe/coKb2/BOibUiT5tWZBQ5UHFu+TBBB8CvXjdSCWJUk2gnwyP4O4dHl9KnOozN5UOgb HFdAj8JFLAiPJsfg0JrJswh/6cK77knU1Y38vuryvAwxkt6eupOipGiYPMNoY08JzXNGchz4Kk2 2Mgb9wuw2FBBi92+QNlFERvmK0nz7I0Qm0YfpcZqz3VyX1FSZsi33u7ZFdJGALxuOUQ== X-Received: by 2002:a17:907:d114:b0:bc6:3181:7711 with SMTP id a640c23a62f3a-bc6318178ddmr382206866b.35.1778156873477; Thu, 07 May 2026 05:27:53 -0700 (PDT) X-Received: by 2002:a17:907:d114:b0:bc6:3181:7711 with SMTP id a640c23a62f3a-bc6318178ddmr382202966b.35.1778156872910; Thu, 07 May 2026 05:27:52 -0700 (PDT) Received: from cluster.. (4f.55.790d.ip4.static.sl-reverse.com. [13.121.85.79]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-bc81cd34ce8sm76552566b.9.2026.05.07.05.27.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2026 05:27:52 -0700 (PDT) From: Alex Markuze To: ceph-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org, idryomov@gmail.com, vdubeyko@redhat.com, Alex Markuze Subject: [PATCH v4 06/11] ceph: add manual reset debugfs control and tracepoints Date: Thu, 7 May 2026 12:27:32 +0000 Message-Id: <20260507122737.2804094-7-amarkuze@redhat.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260507122737.2804094-1-amarkuze@redhat.com> References: <20260507122737.2804094-1-amarkuze@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Add the debugfs and trace plumbing used to trigger and observe manual client reset. The reset interface exposes a trigger file for operator-initiated reset and a status file for tracking the most recent run. The tracepoints record scheduling, completion, and blocked caller behavior so reset progress can be diagnosed from the client side. debugfs layout under /sys/kernel/debug/ceph//reset/: trigger - write to initiate a manual reset status - read to see the most recent reset result The reset directory is cleaned up via debugfs_remove_recursive() on the parent, so individual file dentries are not stored. Tracepoints: ceph_client_reset_schedule - reset queued ceph_client_reset_complete - reset finished (success or failure) ceph_client_reset_blocked - caller blocked waiting for reset ceph_client_reset_unblocked - caller unblocked after reset All tracepoints use a null-safe access for monc.auth->global_id to guard against early-init or late-teardown edge cases. Signed-off-by: Alex Markuze --- fs/ceph/debugfs.c | 103 ++++++++++++++++++++++++++++++++++++ fs/ceph/mds_client.c | 7 +++ fs/ceph/super.h | 1 + include/trace/events/ceph.h | 67 +++++++++++++++++++++++ 4 files changed, 178 insertions(+) diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c index e2463f93cf6b..18eb5da03411 100644 --- a/fs/ceph/debugfs.c +++ b/fs/ceph/debugfs.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include @@ -392,6 +393,90 @@ static int status_show(struct seq_file *s, void *p) return 0; } +static int reset_status_show(struct seq_file *s, void *p) +{ + struct ceph_fs_client *fsc = s->private; + struct ceph_mds_client *mdsc = fsc->mdsc; + struct ceph_client_reset_state *st; + u64 trigger = 0, success = 0, failure = 0; + unsigned long last_start = 0, last_finish = 0; + int last_errno = 0; + enum ceph_client_reset_phase phase = CEPH_CLIENT_RESET_IDLE; + bool drain_timed_out = false; + int sessions_reset = 0; + int blocked_requests = 0; + char reason[CEPH_CLIENT_RESET_REASON_LEN]; + + if (!mdsc) + return 0; + + st = &mdsc->reset_state; + + spin_lock(&st->lock); + trigger = st->trigger_count; + success = st->success_count; + failure = st->failure_count; + last_start = st->last_start; + last_finish = st->last_finish; + last_errno = st->last_errno; + phase = st->phase; + drain_timed_out = st->drain_timed_out; + sessions_reset = st->sessions_reset; + strscpy(reason, st->last_reason, sizeof(reason)); + spin_unlock(&st->lock); + + blocked_requests = atomic_read(&st->blocked_requests); + + seq_printf(s, "phase: %s\n", ceph_reset_phase_name(phase)); + seq_printf(s, "trigger_count: %llu\n", trigger); + seq_printf(s, "success_count: %llu\n", success); + seq_printf(s, "failure_count: %llu\n", failure); + if (last_start) + seq_printf(s, "last_start_ms_ago: %u\n", + jiffies_to_msecs(jiffies - last_start)); + else + seq_puts(s, "last_start_ms_ago: (never)\n"); + if (last_finish) + seq_printf(s, "last_finish_ms_ago: %u\n", + jiffies_to_msecs(jiffies - last_finish)); + else + seq_puts(s, "last_finish_ms_ago: (never)\n"); + seq_printf(s, "last_errno: %d\n", last_errno); + seq_printf(s, "last_reason: %s\n", + reason[0] ? reason : "(none)"); + seq_printf(s, "drain_timed_out: %s\n", + drain_timed_out ? "yes" : "no"); + seq_printf(s, "sessions_reset: %d\n", sessions_reset); + seq_printf(s, "blocked_requests: %d\n", blocked_requests); + + return 0; +} + +static ssize_t reset_trigger_write(struct file *file, const char __user *buf, + size_t len, loff_t *ppos) +{ + struct ceph_fs_client *fsc = file->private_data; + struct ceph_mds_client *mdsc = fsc->mdsc; + char reason[CEPH_CLIENT_RESET_REASON_LEN]; + size_t copy; + int ret; + + if (!mdsc) + return -ENODEV; + + copy = min_t(size_t, len, sizeof(reason) - 1); + if (copy && copy_from_user(reason, buf, copy)) + return -EFAULT; + reason[copy] = '\0'; + strim(reason); + + ret = ceph_mdsc_schedule_reset(mdsc, reason); + if (ret) + return ret; + + return len; +} + static int subvolume_metrics_show(struct seq_file *s, void *p) { struct ceph_fs_client *fsc = s->private; @@ -450,6 +535,7 @@ DEFINE_SHOW_ATTRIBUTE(mdsc); DEFINE_SHOW_ATTRIBUTE(caps); DEFINE_SHOW_ATTRIBUTE(mds_sessions); DEFINE_SHOW_ATTRIBUTE(status); +DEFINE_SHOW_ATTRIBUTE(reset_status); DEFINE_SHOW_ATTRIBUTE(metrics_file); DEFINE_SHOW_ATTRIBUTE(metrics_latency); DEFINE_SHOW_ATTRIBUTE(metrics_size); @@ -521,6 +607,13 @@ static int metric_features_show(struct seq_file *s, void *p) DEFINE_SHOW_ATTRIBUTE(metric_features); +static const struct file_operations ceph_reset_trigger_fops = { + .owner = THIS_MODULE, + .open = simple_open, + .write = reset_trigger_write, + .llseek = noop_llseek, +}; + /* * debugfs */ @@ -554,6 +647,7 @@ void ceph_fs_debugfs_cleanup(struct ceph_fs_client *fsc) debugfs_remove(fsc->debugfs_caps); debugfs_remove(fsc->debugfs_status); debugfs_remove(fsc->debugfs_mdsc); + debugfs_remove_recursive(fsc->debugfs_reset_dir); debugfs_remove(fsc->debugfs_subvolume_metrics); debugfs_remove_recursive(fsc->debugfs_metrics_dir); doutc(fsc->client, "done\n"); @@ -602,6 +696,15 @@ void ceph_fs_debugfs_init(struct ceph_fs_client *fsc) fsc, &caps_fops); + fsc->debugfs_reset_dir = debugfs_create_dir("reset", + fsc->client->debugfs_dir); + debugfs_create_file("trigger", 0200, + fsc->debugfs_reset_dir, fsc, + &ceph_reset_trigger_fops); + debugfs_create_file("status", 0400, + fsc->debugfs_reset_dir, fsc, + &reset_status_fops); + fsc->debugfs_status = debugfs_create_file("status", 0400, fsc->client->debugfs_dir, diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index ce773b1095da..b16638ebff7f 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -5324,6 +5324,7 @@ int ceph_mdsc_wait_for_reset(struct ceph_mds_client *mdsc) blocked_count = atomic_inc_return(&st->blocked_requests); doutc(cl, "request blocked during reset, %d total blocked\n", blocked_count); + trace_ceph_client_reset_blocked(mdsc, blocked_count); retry: remaining = max_t(long, deadline - jiffies, 1); @@ -5334,10 +5335,12 @@ int ceph_mdsc_wait_for_reset(struct ceph_mds_client *mdsc) if (wait_ret == 0) { atomic_dec(&st->blocked_requests); pr_warn_client(cl, "timed out waiting for reset to complete\n"); + trace_ceph_client_reset_unblocked(mdsc, -ETIMEDOUT); return -ETIMEDOUT; } if (wait_ret < 0) { atomic_dec(&st->blocked_requests); + trace_ceph_client_reset_unblocked(mdsc, (int)wait_ret); return (int)wait_ret; /* -ERESTARTSYS */ } @@ -5352,12 +5355,14 @@ int ceph_mdsc_wait_for_reset(struct ceph_mds_client *mdsc) if (time_before(jiffies, deadline)) goto retry; atomic_dec(&st->blocked_requests); + trace_ceph_client_reset_unblocked(mdsc, -ETIMEDOUT); return -ETIMEDOUT; } ret = st->last_errno; spin_unlock(&st->lock); atomic_dec(&st->blocked_requests); + trace_ceph_client_reset_unblocked(mdsc, ret); return ret ? -EAGAIN : 0; } @@ -5387,6 +5392,7 @@ static void ceph_mdsc_reset_complete(struct ceph_mds_client *mdsc, int ret) /* Wake up all requests that were blocked waiting for reset */ wake_up_all(&st->blocked_wq); + trace_ceph_client_reset_complete(mdsc, ret); } static void ceph_mdsc_reset_workfn(struct work_struct *work) @@ -5749,6 +5755,7 @@ int ceph_mdsc_schedule_reset(struct ceph_mds_client *mdsc, pr_info_client(mdsc->fsc->client, "manual session reset scheduled (reason=\"%s\")\n", msg); + trace_ceph_client_reset_schedule(mdsc, msg); return 0; } diff --git a/fs/ceph/super.h b/fs/ceph/super.h index a4993644d543..1d6aab060780 100644 --- a/fs/ceph/super.h +++ b/fs/ceph/super.h @@ -179,6 +179,7 @@ struct ceph_fs_client { struct dentry *debugfs_status; struct dentry *debugfs_mds_sessions; struct dentry *debugfs_metrics_dir; + struct dentry *debugfs_reset_dir; struct dentry *debugfs_subvolume_metrics; #endif diff --git a/include/trace/events/ceph.h b/include/trace/events/ceph.h index 08cb0659fbfc..1b990632f62b 100644 --- a/include/trace/events/ceph.h +++ b/include/trace/events/ceph.h @@ -226,6 +226,73 @@ TRACE_EVENT(ceph_handle_caps, __entry->mseq) ); +/* + * Client reset tracepoints - identify the client by its monitor- + * assigned global_id so traces remain meaningful when kernel pointer + * hashing is enabled. + */ +TRACE_EVENT(ceph_client_reset_schedule, + TP_PROTO(const struct ceph_mds_client *mdsc, const char *reason), + TP_ARGS(mdsc, reason), + TP_STRUCT__entry( + __field(u64, client_id) + __string(reason, reason ? reason : "") + ), + TP_fast_assign( + __entry->client_id = mdsc->fsc->client->monc.auth ? + mdsc->fsc->client->monc.auth->global_id : 0; + __assign_str(reason); + ), + TP_printk("client_id=%llu reason=%s", + __entry->client_id, __get_str(reason)) +); + +TRACE_EVENT(ceph_client_reset_complete, + TP_PROTO(const struct ceph_mds_client *mdsc, int ret), + TP_ARGS(mdsc, ret), + TP_STRUCT__entry( + __field(u64, client_id) + __field(int, ret) + ), + TP_fast_assign( + __entry->client_id = mdsc->fsc->client->monc.auth ? + mdsc->fsc->client->monc.auth->global_id : 0; + __entry->ret = ret; + ), + TP_printk("client_id=%llu ret=%d", __entry->client_id, __entry->ret) +); + +TRACE_EVENT(ceph_client_reset_blocked, + TP_PROTO(const struct ceph_mds_client *mdsc, int blocked_count), + TP_ARGS(mdsc, blocked_count), + TP_STRUCT__entry( + __field(u64, client_id) + __field(int, blocked_count) + ), + TP_fast_assign( + __entry->client_id = mdsc->fsc->client->monc.auth ? + mdsc->fsc->client->monc.auth->global_id : 0; + __entry->blocked_count = blocked_count; + ), + TP_printk("client_id=%llu blocked_count=%d", __entry->client_id, + __entry->blocked_count) +); + +TRACE_EVENT(ceph_client_reset_unblocked, + TP_PROTO(const struct ceph_mds_client *mdsc, int ret), + TP_ARGS(mdsc, ret), + TP_STRUCT__entry( + __field(u64, client_id) + __field(int, ret) + ), + TP_fast_assign( + __entry->client_id = mdsc->fsc->client->monc.auth ? + mdsc->fsc->client->monc.auth->global_id : 0; + __entry->ret = ret; + ), + TP_printk("client_id=%llu ret=%d", __entry->client_id, __entry->ret) +); + #undef EM #undef E_ #endif /* _TRACE_CEPH_H */ -- 2.34.1