given a pointer to xfs_inode_t, how to determine path?

given a pointer to xfs_inode_t, how to determine path?

[Chris Friesen]

Hi,

We're running 2.6.27 (upgrading not currently possible, embedded product).

We had a situation arise where we could see in stack traces that a 
number of tasks were stuck in vn_iowait().  That function takes a 
pointer to xfs_inode_t.  Given that, would it be possible to work 
backwards to determine a filesystem path corresponding to that inode?  I 
realize it would likely only go back to the head of the filesystem, but 
that would be fine.

Thanks,
Chris


-- 

Chris Friesen
Software Designer

3500 Carling Avenue
Ottawa, Ontario K2H 8E9
www.genband.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: given a pointer to xfs_inode_t, how to determine path?

[Ben Myers]

Hi Chris,

On Mon, Sep 17, 2012 at 09:51:37AM -0600, Chris Friesen wrote:
> We're running 2.6.27 (upgrading not currently possible, embedded product).
>
> We had a situation arise where we could see in stack traces that a
> number of tasks were stuck in vn_iowait().  That function takes a
> pointer to xfs_inode_t.  Given that, would it be possible to work
> backwards to determine a filesystem path corresponding to that
> inode?  I realize it would likely only go back to the head of the
> filesystem, but that would be fine.

Yep, it's possible.

echo t > /proc/sysrq-trigger won't give you the full stack, but you may be able
to get an idea of what is going on, well enough to make some guesses as to the
culprit.  That's probably your best first step.

If you have crash installed:

1) disassemble vn_iowait and find out which register the xfs_inode_t pointer is in.
2) print out the full stacks of all the processes in the system. 'bt -f', I think.
3) grep for the xfs_inode_t pointer on those stacks.

Regards,
	Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: given a pointer to xfs_inode_t, how to determine path?

[Dave Chinner]

On Mon, Sep 17, 2012 at 11:21:20AM -0500, Ben Myers wrote:
> Hi Chris,
> 
> On Mon, Sep 17, 2012 at 09:51:37AM -0600, Chris Friesen wrote:
> > We're running 2.6.27 (upgrading not currently possible, embedded product).
> >
> > We had a situation arise where we could see in stack traces that a
> > number of tasks were stuck in vn_iowait().  That function takes a
> > pointer to xfs_inode_t.  Given that, would it be possible to work
> > backwards to determine a filesystem path corresponding to that
> > inode?  I realize it would likely only go back to the head of the
> > filesystem, but that would be fine.
> 
> Yep, it's possible.
> 
> echo t > /proc/sysrq-trigger won't give you the full stack, but you may be able
> to get an idea of what is going on, well enough to make some guesses as to the
> culprit.  That's probably your best first step.

Doesn't give you the path of the inode, or anything that can tell
you what the inode number is so you could run "find <dir> -inum
<num>" after the system has recovered.

> If you have crash installed:
> 
> 1) disassemble vn_iowait and find out which register the xfs_inode_t pointer is in.
> 2) print out the full stacks of all the processes in the system. 'bt -f', I think.

'bt -F <pid>' tells you what entries on the stack are of a
particular symbol. i.e. they'll show up as [xfs_inode] instead of a
number for all addresses from inside the xfs inode slab cache.
comapring the output of 'bt -F' and 'bt -f' will get you the inode
address without needing to be able to read x86-64 assembly
language...

> 3) grep for the xfs_inode_t pointer on those stacks.

Which doesn't get you the path, either. From there, follow to the
struct inode, and from there to the struct dentry and grab the file
name, then walk the d_parent dentry links back up to the root and
construct the path that way.

Alternatively, I think you can walk from the top down by looking
that the open files that a given process has and working down to the
xfs_inode, but it's been a long time since I've had to come from
that direction. Indeed, you might even be able to use lsof to do
this and get an idea from userspace what files are currently
open on the hung processes.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: given a pointer to xfs_inode_t, how to determine path?

[Ben Myers]

Hey,

On Tue, Sep 18, 2012 at 08:49:57AM +1000, Dave Chinner wrote:
> On Mon, Sep 17, 2012 at 11:21:20AM -0500, Ben Myers wrote:
> > Hi Chris,
> > 
> > On Mon, Sep 17, 2012 at 09:51:37AM -0600, Chris Friesen wrote:
> > > We're running 2.6.27 (upgrading not currently possible, embedded product).
> > >
> > > We had a situation arise where we could see in stack traces that a
> > > number of tasks were stuck in vn_iowait().  That function takes a
> > > pointer to xfs_inode_t.  Given that, would it be possible to work
> > > backwards to determine a filesystem path corresponding to that
> > > inode?  I realize it would likely only go back to the head of the
> > > filesystem, but that would be fine.
> > 
> > Yep, it's possible.
> > 
> > echo t > /proc/sysrq-trigger won't give you the full stack, but you may be able
> > to get an idea of what is going on, well enough to make some guesses as to the
> > culprit.  That's probably your best first step.
> 
> Doesn't give you the path of the inode, or anything that can tell
> you what the inode number is so you could run "find <dir> -inum
> <num>" after the system has recovered.
> 
> > If you have crash installed:
> > 
> > 1) disassemble vn_iowait and find out which register the xfs_inode_t pointer is in.
> > 2) print out the full stacks of all the processes in the system. 'bt -f', I think.
> 
> 'bt -F <pid>' tells you what entries on the stack are of a
> particular symbol. i.e. they'll show up as [xfs_inode] instead of a
> number for all addresses from inside the xfs inode slab cache.
> comapring the output of 'bt -F' and 'bt -f' will get you the inode
> address without needing to be able to read x86-64 assembly
> language...
> 
> > 3) grep for the xfs_inode_t pointer on those stacks.
> 
> Which doesn't get you the path, either. From there, follow to the
> struct inode, and from there to the struct dentry and grab the file
> name, then walk the d_parent dentry links back up to the root and
> construct the path that way.
> 
> Alternatively, I think you can walk from the top down by looking
> that the open files that a given process has and working down to the
> xfs_inode, but it's been a long time since I've had to come from
> that direction. Indeed, you might even be able to use lsof to do
> this and get an idea from userspace what files are currently
> open on the hung processes.

Yep, I misunderstood the question.  I didn't mean to mislead you Chris!
Really!  ;)

-Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs