cluster-devel.redhat.com archive mirror
 help / color / mirror / Atom feed
From: Bob Peterson <rpeterso@redhat.com>
To: cluster-devel.redhat.com
Subject: [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free
Date: Fri, 1 Feb 2019 09:51:21 -0500 (EST)	[thread overview]
Message-ID: <1750009173.69290319.1549032681784.JavaMail.zimbra@redhat.com> (raw)
In-Reply-To: <9b9dd4a6-6529-98e9-b1a1-aab8426cfa86@citrix.com>

Hi Ross,

----- Original Message -----
> Do you have any suggestions for tracking down the root cause?

One time, when I had a similar problem in rhel7, and couldn't use
kernel tracing because there were millions of glocks involved.
The trace was too huge and quickly swamped the biggest possible
kernel trace buffer. So I ended up writing this ugly, hacky patch
that's attached. Perhaps you can use it as a starting point.

The idea is: every time there's a get or a put to a glock, it
saves off a 1-byte identifier of what function did the get/put.
It saved it in a new 64-byte field kept for each glock, which of
course meant the slab became much bigger, but it was never meant
to be shipped, right?

Then, when the problem occurred, it would dump out the problematic
glock, including the 64-byte get/put history value.
Then I would go through it and identify the history of what went
wrong.

Since this is a fairly old (2015) patch that targets an old rhel7,
it will obviously need a lot of updating to get it to work, but
it might work better than the kernel tracing, depending on how
many glocks are involved in your test.

Regards,

Bob Peterson
Red Hat File Systems
-------------- next part --------------
A non-text attachment was scrubbed...
Name: get_put.patch
Type: text/x-patch
Size: 25569 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20190201/4bb1e1c2/attachment.bin>

  parent reply	other threads:[~2019-02-01 14:51 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-01-31 10:55 [Cluster-devel] [PATCH 0/2] GFS2 counting fixes Ross Lagerwall
2019-01-31 10:55 ` [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free Ross Lagerwall
2019-01-31 11:23   ` Steven Whitehouse
2019-01-31 14:40   ` Bob Peterson
2019-01-31 17:18   ` Andreas Gruenbacher
2019-02-01  9:23     ` Ross Lagerwall
2019-02-01 14:34       ` Bob Peterson
2019-02-01 14:51       ` Bob Peterson [this message]
2019-02-01 15:03       ` [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free (Another debug patch) Bob Peterson
2019-03-26 18:49     ` [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free Ross Lagerwall
2019-03-26 19:14       ` Bob Peterson
2019-04-01 22:59         ` Andreas Gruenbacher
2019-04-05 17:50           ` Andreas Gruenbacher
2019-04-09 15:36             ` Ross Lagerwall
2019-04-09 15:41               ` Andreas Gruenbacher
2019-01-31 10:55 ` [Cluster-devel] [PATCH 2/2] gfs2: Fix lru_count going negative Ross Lagerwall
2019-01-31 11:21   ` Steven Whitehouse
2019-01-31 14:36   ` Bob Peterson
2019-01-31 15:04     ` Bob Peterson
2019-01-31 15:23     ` Ross Lagerwall
2019-01-31 18:32   ` Andreas Gruenbacher

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1750009173.69290319.1549032681784.JavaMail.zimbra@redhat.com \
    --to=rpeterso@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).