Re: PROBLEM: Possible race between xen, md, dm and/or xfs

[Dave Chinner <david@fromorbit.com>]

On Fri, Jun 08, 2012 at 03:45:46PM +1000, Jason Stubbs wrote:
> Hi,
> 
> To quickly summarize, on a Xen domU instance with a disk structure of XFS on
> LVM2 on RAID10 on 8x virtual disks, all tasks performing I/O to said XFS
> partition hung and I cannot prove or disprove it to be dom0 issue.
> 
> And now the long(er) version:
> 
> On an Amazon EC2 (xen) instance, I had I/O to one of the EBS (Elastic Block
> Store virtual disk) devices block with iostat showing one single request
> pending. Kernel logs showed hung tasks so after grabbing those I reset the
> instance but - while I'm told that Amazon's logs don't show any problems
> with the EBS - Amazon want the opportunity to exclude an EBS problem by
> examining things from the dom0 side while the problem is occurring before
> delving into the kernel.

Yup, everything is hung waiting for that one IO to complete. Nothing
wrong with MD, LVM, or XFS. The problem is either that EBS never
completed the IO, or Xen swallowed it and it never made to it to the
guest OS. Either way, it does not appear to be a problem in the
higher levels of the linux storage stack.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com