All of lore.kernel.org
 help / color / mirror / Atom feed
From: "J. Hart" <jfhart085@gmail.com>
To: Christoph Hellwig <hch@lst.de>
Cc: linux-nvme@lists.infradead.org, kbusch@kernel.org, axboe@fb.com,
	sagi@grimberg.me, jfhart085@gmail.com
Subject: Re: nvme nvme0: I/O 0 (I/O Cmd) QID 1 timeout, aborting, source drive corruption observed
Date: Thu, 15 Dec 2022 18:07:32 +0900	[thread overview]
Message-ID: <d4c4eea9-e573-df0c-cca5-536bcff6c0a9@gmail.com> (raw)
In-Reply-To: <20221215082344.GB3816@lst.de>

On 12/15/22 5:23 PM, Christoph Hellwig wrote:
>>
>> dmesg reports the following:
> 
> nvme0 is the destination driver I guess?

That is correct.  The device /dev/nvme0n1p3 is the destination partition 
on the NVME drive, and was mounted at /mnt/root_new.

>> [Dec14 19:24] nvme nvme0: I/O 0 (I/O Cmd) QID 1 timeout, aborting
> 
> Can you enable CONFIG_NVME_VERBOSE_ERRORS so that we can see what
> commands are hanging?

I will do this and run the requisite testing right away.  I'll reply 
with the results as soon as I have them.

>> I have also observed file system corruption on the source drive of the
>> transfer.  I would not normally think this to be related, except that after
>> the first time I observed it, I made certain that I corrected the file
>> content before any additional attempts, but have seen this again after
>> every attempt.  The modification dates and file sizes did not change, but
>> the file content on the source drive did.  I confirmed this using the
>> "diff" utility, and again using a rsync dry run with the check sum test
>> enabled.

I should also note that I did a third test using md5sum and confirmed 
that the sums obtained thereby were different.

> Ok, that's really odd.  The only way I could think of that happening
> is if the driver does stay DMAs, which would be really grave.

My apologies....I am not sure what is meant by "stay DMAs".  Is there 
something I can look for here ?

> Do you have CONFIG_INTEL_IOMMU and CONFIG_INTEL_IOMMU_DEFAULT_ON enabled?
> If not, it would be good to enable those to see if the iommu catches
> any stray DMAs.

I will enable these as well and reply with the test results.  Thanks 
very much for your generous assistance.

J. Hart


  reply	other threads:[~2022-12-15  9:07 UTC|newest]

Thread overview: 27+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-12-15  1:38 nvme nvme0: I/O 0 (I/O Cmd) QID 1 timeout, aborting, source drive corruption observed J. Hart
2022-12-15  8:23 ` Christoph Hellwig
2022-12-15  9:07   ` J. Hart [this message]
2022-12-15  9:09     ` Christoph Hellwig
2022-12-15  9:15       ` J. Hart
2022-12-15 13:33       ` J. Hart
2022-12-15 17:34         ` Keith Busch
2022-12-15 22:30           ` J. Hart
2022-12-16  6:39             ` Christoph Hellwig
2022-12-16 19:08               ` Keith Busch
2023-01-18 10:27             ` Mark Ruijter
2022-12-16 23:16 ` Keith Busch
2022-12-17  1:28   ` J. Hart
2022-12-19 14:41     ` Keith Busch
2022-12-20  1:10       ` J. Hart
2022-12-20 16:56         ` Keith Busch
2022-12-21  7:50           ` Christoph Hellwig
  -- strict thread matches above, loose matches on Subject: below --
2022-12-17 12:07 J. Hart
2022-12-17 15:07 J. Hart
2022-12-17 16:14 J. Hart
2022-12-17 21:57 J. Hart
2022-12-18  6:20 J. Hart
2022-12-18 12:08 J. Hart
2022-12-19 14:45 ` Keith Busch
2022-12-19 23:40   ` J. Hart
2022-12-20 18:10     ` Keith Busch
2022-12-20 14:04   ` J. Hart

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d4c4eea9-e573-df0c-cca5-536bcff6c0a9@gmail.com \
    --to=jfhart085@gmail.com \
    --cc=axboe@fb.com \
    --cc=hch@lst.de \
    --cc=kbusch@kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.